Destructive operation on submodules with submodule.recurse = true git setting

[a666]

describe your issue

I added pre-commit to an ansible playbook repository that does extensive usage of git submodules for roles.

I had two submodules with extensive changes:

$ gst
On branch master
Your branch is ahead of 'origin/master' by 3 commits.
  (use "git push" to publish your local commits)
...
        modified:   roles/dnscrypt-proxy (modified content) <-- submodule with extensive (non-commited) changes
        modified:   roles/syncthing (modified content) <-- another submodule with extensive (non-commited) changes
...

Upon doing a commit I received an error (check ~/.cache/pre-commit/pre-commit.log section) due to having some problems in my repo and its submodules.
Notice that I cold not even run git status afterwards:

$ gst
fatal: not a git repository: roles/matomo/../../.git/modules/roles/matomo

After fixing the problems:

$ rm -rf roles/matomo/ .git/modules/roles/matomo/
$ git submodule update
Cloning into '/home/apoc/w/posg-ops/roles/matomo'...
Submodule path 'roles/matomo': checked out '######'
fatal: could not get a repository handle for submodule 'roles/php'
$ rm -rf roles/php/ .git/modules/roles/php
$ git submodule update
Cloning into '/home/apoc/w/posg-ops/roles/php'...
Submodule path 'roles/php': checked out '######'

and running git apply to have my changes back:

$ git apply /home/a666/.cache/pre-commit/patch1632709290-1076629

I noticed that my two submodules now had no changes in them...
And unfortunately I see nothing in pre-commit cache area to help me recover them.

pre-commit --version

pre-commit 2.15.0

.pre-commit-config.yaml

---
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.0.1
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-toml
      - id: check-added-large-files

  - repo: https://github.com/editorconfig-checker/editorconfig-checker.python
    rev: 2.3.54
    hooks:
      - id: editorconfig-checker

  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.51.0
    hooks:
      - id: terraform_validate
      - id: terraform_fmt
      - id: checkov
      - id: terrascan
      - id: terraform_tfsec

~/.cache/pre-commit/pre-commit.log (if present)

version information

pre-commit version: 2.15.0
sys.version:
    3.9.7 (default, Aug 31 2021, 13:28:12)
    [GCC 11.1.0]
sys.executable: /home/a666/.local/pipx/venvs/pre-commit/bin/python
os.name: posix
sys.platform: linux

error information

An unexpected error has occurred: CalledProcessError: command: ('/usr/lib/git-core/git', 'checkout', '--', '.')
return code: 255
expected return code: 0
stdout: (none)
stderr:
    Migrating git directory of 'roles/fail2ban' from
    '/home/a666/ops/roles/fail2ban/.git' to
    '/home/a666/ops/.git/modules/roles/fail2ban'
    fatal: not a git repository: ../../.git/modules/roles/matomo
    error: Submodule 'roles/matomo' could not be updated.
    Migrating git directory of 'roles/openproject' from
    '/home/a666/ops/roles/openproject/.git' to
    '/home/a666/ops/.git/modules/roles/openproject'
    Migrating git directory of 'roles/oryhydra' from
    '/home/a666/ops/roles/oryhydra/.git' to
    '/home/a666/ops/.git/modules/roles/oryhydra'
    fatal: not a git repository: ../../.git/modules/roles/php
    error: Submodule 'roles/php' could not be updated.

Traceback (most recent call last):
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/error_handler.py", line 65, in error_handler
    yield
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/main.py", line 368, in main
    return hook_impl(
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/commands/hook_impl.py", line 237, in hook_impl
    return retv | run(config, store, ns)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/commands/run.py", line 399, in run
    exit_stack.enter_context(staged_files_only(store.directory))
  File "/usr/lib/python3.9/contextlib.py", line 448, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/staged_files_only.py", line 92, in staged_files_only
    with _intent_to_add_cleared(), _unstaged_changes_cleared(patch_dir):
  File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/staged_files_only.py", line 61, in _unstaged_changes_cleared
    cmd_output_b('git', 'checkout', '--', '.', env=no_checkout_env)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/util.py", line 154, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/usr/lib/git-core/git', 'checkout', '--', '.')
return code: 255
expected return code: 0
stdout: (none)
stderr:
    Migrating git directory of 'roles/fail2ban' from
    '/home/a666/ops/roles/fail2ban/.git' to
    '/home/a666/ops/.git/modules/roles/fail2ban'
    fatal: not a git repository: ../../.git/modules/roles/matomo
    error: Submodule 'roles/matomo' could not be updated.
    Migrating git directory of 'roles/openproject' from
    '/home/a666/ops/roles/openproject/.git' to
    '/home/a666/ops/.git/modules/roles/openproject'
    Migrating git directory of 'roles/oryhydra' from
    '/home/a666/ops/roles/oryhydra/.git' to
    '/home/a666/ops/.git/modules/roles/oryhydra'
    fatal: not a git repository: ../../.git/modules/roles/php
    error: Submodule 'roles/php' could not be updated.

[asottile]

there's a test in our testsuite which shows this works correctly -- do you have more information on how to reproduce this?

looking at your output it looks like your submodules may have already been broken since git was unable to clean them manage them properly?

your report is also incomplete -- you have not shared the command you ran and its output so I cannot help you further without that

[asottile]

also, do you have any post-checkout hooks installed?

[a666]

looking at your output it looks like your submodules may have already been broken since git was unable to clean them manage them properly?

There was definitively a problem with them, but the changes to these submodules date from months back, there's around 20 commits following those changes.

your report is also incomplete -- you have not shared the command you ran and its output so I cannot help you further without that

From the history, it was a simple git commit -v.
Unfortunately I no longer have the pane history due to machine reboot, the error displayed was very similar to the error information section, sorry about that.

also, do you have any post-checkout hooks installed?

None.

[asottile]

without a reproduction there isn't really much I can fix here :( -- the operations run are supposed to be safe (especially because --recurse-submodules is not passed) so I can only guess there's some bug with git itself that caused your issue

[asottile]

ahah -- is it possible you have this setting enabled? https://git-scm.com/docs/git-config#Documentation/git-config.txt-submodulerecurse

let me see if that's what can cause this ?

[a666]

Well, apparently the last sysadmin decided to put it in the config, and I ended getting it too after rsyncking the repo from his account...

$ git config --list --show-origin | rg recurse
file:.git/config        submodule.recurse=true

[asottile]

rough! well I'll at least be able to fix that by ignoring that setting for pre-commit's programmatic uses of git

[a666]

So if I understand well, having submodule.recurse=true made pre-commit run a git checkout . inside the submodules?

[asottile]

submodule.recurse = 1 makes git checkout -- . also run git checkout -- . in each submodule -- seems like a really bad git feature

[a666]

Well, submodules were always like a weird git workaround to a difficult problem (I think) and not very though out...