git apply error when using git-crypt

[ranimufid]

Hi. I am currently using git-crypt in tandem with some pre-commit hooks

Upon adding a new file for encryption --> a corresponding .gitattributes entry --> git commit -m "blah" I encounter the following error:

[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to /Users/bob/.cache/pre-commit/patch1530518673.
Trim Trailing Whitespace.................................................Passed
Check Yaml...............................................................Passed
Fix End of Files.........................................................Passed
Ansible-lint.............................................................Passed
[WARNING] Stashed changes conflicted with hook auto-fixes... Rolling back fixes...
An unexpected error has occurred: CalledProcessError: Command: ('/usr/local/Cellar/git/2.18.0/libexec/git-core/git', '-c', 'core.autocrlf=false', 'apply', '--whitespace=nowarn', '/Users/bob/.cache/pre-commit/patch1530518673')
Return code: 128
Expected return code: 0
Output: (none)
Errors:
    error: unrecognized input

git version: 2.18.0
pre-commit version: 1.10.2
.pre-commit-config.yaml

repos:
-   repo: git://github.com/pre-commit/pre-commit-hooks
    rev: v1.3.0
    hooks:
    -   id: trailing-whitespace
    -   id: check-yaml
    -   id: end-of-file-fixer
-   repo: https://github.com/willthames/ansible-lint.git
    rev: v3.4.23
    hooks:
    - id: ansible-lint
      entry: ansible-lint bootstrap.yml

.gitattributes:

hosts/testing/secret-vars.yml filter=git-crypt diff=git-crypt

[asottile]

Thanks for the report! could you provide some more information (if possible) for me?

it appears from the output that you committed with an unstaged change, to which file was that and if possible could you share the patch file mentioned?

I suspect the diff=git-crypt is somehow throwing off the way we generate and then reapply patches and will need to be worked around.

[ranimufid]

thanks for your prompt response @asottile.

Context:
I'm trying to encrypt some ansible secrets using git-crypt. I'm also doing ansible-lint as a pre-commit hook.

it appears from the output that you committed with an unstated change

The change which triggered the error is creating an empty file /hosts/testing/secret-vars.yml with a corresponding entry in .gitattributes

.gitattributes

hosts/testing/secret-vars.yml filter=git-crypt diff=git-crypt

to which file was that and if possible could you share the patch file mentioned

The patch file corresponding to the above change

diff --git a/hosts/testing/secret-vars.yml b/hosts/testing/secret-vars.yml
GIT binary patch
literal 0
HcmV?d00001

literal 22
saefs@_sfdsfsdfsdf?>t6Gdv(psdfsdfsdfsf345`ISOHcp2nGNE

Do let me know if you need further info

[asottile]

ok, I have a reproduction:

#!/usr/bin/env bash
set -euxo pipefail

rm -rf repo

git init repo
cd repo
git commit --allow-empty -m 'Initial empty commit'
git-crypt init

cat > .pre-commit-config.yaml << EOF
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v1.3.0
    hooks:
    -   id: trailing-whitespace
EOF
git add .pre-commit-config.yaml
pre-commit install
git commit -m "Add pre-commit config"

echo 'f filter=git-crypt diff=git-crypt' > .gitattributes
git add .gitattributes
echo 'hello world' > f
git add f
rm f && touch f

git commit -m "test"

which ends with:

+ git commit -m test
[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to /home/asottile/.cache/pre-commit/patch1530539647.
Trim Trailing Whitespace.................................................Passed
[WARNING] Stashed changes conflicted with hook auto-fixes... Rolling back fixes...
An unexpected error has occurred: CalledProcessError: Command: ('/usr/lib/git-core/git', '-c', 'core.autocrlf=false', 'apply', '--whitespace=nowarn', '/home/asottile/.cache/pre-commit/patch1530539647')
Return code: 1
Expected return code: 0
Output: (none)
Errors: 
    error: binary patch to 'f' creates incorrect result (expecting 9dc8b7b3fd7e85892019cbf77d888ce9d9f37b15, got e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
    error: f: patch does not apply
    

Check the log at /home/asottile/.cache/pre-commit/pre-commit.log

Almost certainly due to the smudge filter though I'm not exactly sure what the correct approach is here -- for now you can work around this by making sure you stage the encrypted files before committing.

[ranimufid]

This is how i've worked around it for the time being:

touch hosts/production/secret-vars.yml
echo "#prod vars" >> hosts/production/secret-vars.yml
git add hosts/production/secret-vars.yml
git commit -m "add production secret vars"

echo 'hosts/production/secret-vars.yml filter=git-crypt diff=git-crypt' >> .gitattributes
git add .gitattributes
git commit -m "add production secret vars"

git push

[asottile]

hmm, my crash is actually different from yours -- I wonder there's two issues here :S

[brlin-tw]

I encountered a similar issue as well, here's the log. This is reproduced in some repos that have implemented some clean filters, with some test cases files intentionally committed uncleaned for testing.

[marcjay]

Wondering if there was an update on this issue? If there are any hints on what the issue might be, I'd be keen to offer help fixing - it's something we run into frequently

[asottile]

I started looking into this from the git side but never got a definitive response -- it comes down to unstaged things in a smudge filter generating bogus git diff that cannot be later git applyd

the current known workaround is to make sure such files are git added before committing, but isn't ideal

I'm not sure what it would take to fix this, I suspect a fix from git is necessary (or some workaround where pre-commit identifies smudge-filter files and handles them separately)

[marcjay]

Many thanks @asottile, that all sounds very non-trivial

[asottile]

yeah :'( that's why I haven't dug into it yet beyond the repro above

[yuklia]

Hi! I've got the same issue:

git version 2.24.3 (Apple Git-128)
MacOs BigSur 11.1
git-crypt 0.6.0
pre-commit 2.11.1

pre-commit run --show-diff-on-failure terraform_validate
[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to /Users/yulia/.cache/pre-commit/patch1616497389.
Terraform validate.......................................................Passed
[WARNING] Stashed changes conflicted with hook auto-fixes... Rolling back fixes...
An unexpected error has occurred: CalledProcessError: command: ('/usr/bin/git', '-c', 'core.autocrlf=false', 'apply', '--whitespace=nowarn', '/Users/yulia/.cache/pre-commit/patch1616497389')
return code: 128
expected return code: 0
stdout: (none)
stderr:
    error: unrecognized input
    

.pre-commit-config.yml

repos:
  - repo: git://github.com/antonbabenko/pre-commit-terraform
    rev: v1.46.0
    hooks:
      - id: terraform_fmt
      - id: terraform_docs
      # - id: terraform_tflint
      #   args:
      #     - '--args=--only=terraform_deprecated_interpolation'
      #     - '--args=--only=terraform_deprecated_index'
      #     - '--args=--only=terraform_unused_declarations'
      #     - '--args=--only=terraform_comment_syntax'
      #     - '--args=--only=terraform_documented_outputs'
      #     - '--args=--only=terraform_documented_variables'
      #     - '--args=--only=terraform_typed_variables'
      #     - '--args=--only=terraform_module_pinned_source'
      #     - '--args=--only=terraform_naming_convention'
      #     - '--args=--only=terraform_required_version'
      #     - '--args=--only=terraform_required_providers'
      #     - '--args=--only=terraform_standard_module_structure'
      #     - '--args=--only=terraform_workspace_remote'
  - repo: git://github.com/pre-commit/pre-commit-hooks
    rev: v3.4.0
    hooks:
      - id: check-merge-conflict
  - repo: git://github.com/gruntwork-io/pre-commit
    rev: v0.1.12
    hooks:
      - id: terraform-validate

[yuklia]

it solved for me

as it turned out there were a couple of empty files. I took a look inside /Users/yulia/.cache/pre-commit/pre-commit.log and noticed that 1 commit ago I committed 2 totally empty secrets files. So I filled them in.
then

0 git add .
1 pre-commit install
2 git commit -m 'Your commit msg'
Everything works as expected :))

Terraform fmt............................................................Passed
Terraform docs.......................................(no files to check)Skipped
Terraform validate.......................................................Passed
Check for merge conflicts................................................Passed
git-crypt-check-status...................................................Passed

[philipsd6]

I can confirm that this issue was resolved for me by making sure I wasn't staging empty files that matched my .gitattributes git-crypt smudge/filter settings.

[asottile]

interestingly, adding --textconv to the git diff command makes (at least) my reproduction above succeed. it certainly seems like the wrong thing to do but maybe that's ok?

[asottile]

I submitted a more thorough investigation to the git mailing list -- https://marc.info/?l=git&m=166733577332408&w=2