repo fails git fsck

[zackschuster]

repro:

$ git config --system receive.fsckObjects true
$ git clone git@github.com:jashkenas/underscore.git  # or `git submodule add -- git@github.com:jashkenas/underscore.git`
Cloning into 'underscore'...
remote: Enumerating objects: 215, done.
remote: Counting objects: 100% (215/215), done.
remote: Compressing objects: 100% (130/130), done.
error: object b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3: zeroPaddedFilemode: contains zero-padded file modes
fatal: fsck error in packed object

[jgonggrijp]

@zackschuster Thanks for reaching out.

I had to look online for further information, because I had never heard about git fsck or this particular error before. I eventually found in pallets/flask#2029 that there are two possible solutions for this type of problem. The first really solves the issue by rebuilding the repository, but it is too destructive because all hashes after the faulty object would change. The second amounts to adding some documentation to warn people about the issue and to instruct them how to work around it (which seems perfectly doable). Only the latter solution would be viable for this repo.

All of this seems a bit moot to me, though, because I cannot reproduce the error. I tried running the exact commands from the opening post using Git 2.26.2, except with sudo in front of the first command since this is required in order to change the system-wide settings. The repo just cloned without any errors. I also couldn't find any object with the b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3 hash.

@zackschuster Please double-check that the problem you ran into is not the result of a faulty network transfer, a faulty disk read, or a bug in some particular version of Git. I will close this issue now; if you find additional steps that I can perform in order to reproduce the error after all, or if somebody else can confirm that this is really an issue with the Underscore repo, we will of course reopen it.

[zackschuster]

@jgonggrijp i'm using git 2.29.2 on macOS, compiled in the standard way with autoconf & make. apologies for the error in the console command; truthfully, i have receive.fsckObjects set from $HOME/.gitconfig.

i'm not sure how to track down blob errors on my end, but i'll try to get more details 😄

[zackschuster]

@jgonggrijp i found some info. of note:

1. running git clone --config transfer.fsckobjects=false git@github.com:jashkenas/underscore.git avoids the error (note the transfer.fsckobjects; i had the wrong setting in my original report. apologies!)
2. subsequently, running git fsck in the repo (using v2.29.2) reveals two zeroPaddedFilemode errors
3. git cat-file -p shows a list of files, which suggests they're bad commits; git cat-file -t marks them as type tree. git show output is a less-informative version of git cat-file -p. git log returns nothing.

i've included the console output for reference:

~/Source
$ git clone --config transfer.fsckobjects=false git@github.com:jashkenas/underscore.git
Cloning into 'underscore'...
remote: Enumerating objects: 215, done.
remote: Counting objects: 100% (215/215), done.
remote: Compressing objects: 100% (130/130), done.
remote: Total 10434 (delta 185), reused 101 (delta 85), pack-reused 10219
Receiving objects: 100% (10434/10434), 5.61 MiB | 9.21 MiB/s, done.
Resolving deltas: 100% (6770/6770), done.

~/Source
$ cd underscore 

~/Source/underscore
$ git fsck
Checking object directories: 100% (256/256), done.
warning in tree b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3: zeroPaddedFilemode: contains zero-padded file modes
warning in tree 8a654ec549a230dcab28e54c6e53c460cb9e190d: zeroPaddedFilemode: contains zero-padded file modes
Checking objects: 100% (10434/10434), done.

~/Source/underscore
$ git cat-file -p b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3
100644 blob d8cef485ec7309b938b4c3bc19171458e9139811	LICENSE
100644 blob 333b13f45b62427fd0840c77bb55bfc8fb96be6b	README
100644 blob 721c294b3ddb1d3e6ed35a7d9336e162aab01bb9	Rakefile
100644 blob bdaef73ca2ca82b90d37cec4b9c38606b440d619	index.html
100644 blob 6771de07ecd088853df1376f3693e81584ec2560	package.json
040000 tree d0a277f748b309f3d26e13d1cb5133ff003c468d	test
100644 blob 7cfbf993705f83e7fde17e9745b4187f7a4047b2	underscore-min.js
100644 blob 625411233e487b5cd8da1a32a69e5693b1862c09	underscore.js

~/Source/underscore
$ git cat-file -p 8a654ec549a230dcab28e54c6e53c460cb9e190d
100644 blob 9644b34c3f5c16753d1061497aa7aab426e6c896	LICENSE
100644 blob e41015cd7cc04a940ac4e7a451aeb0b0b384c3a3	README
100644 blob 6af14bb6d570698990d30aa0b46e398ae3fd48b1	Rakefile
100644 blob 1a03be19fc8ddf61b113f71c19d6892fb74ff4f5	index.html
100644 blob 2cbc8128d9db1feabc44a3be14c504e40b6b7e2e	package.json
040000 tree fae9dcb01827d13b2b5fbd9afab07489b249dab8	test
100644 blob 2c8b499882c2a5d662ea35c80ec7ea0fa111814f	underscore-min.js
100644 blob 42bf31a27545d0d5460c655d13ce9bbf76e540b3	underscore.js

~/Source/underscore
$ git cat-file -t b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3
tree

~/Source/underscore
$ git cat-file -t 8a654ec549a230dcab28e54c6e53c460cb9e190d
tree

~/Source/underscore
$ git show b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3
tree b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3

LICENSE
README
Rakefile
index.html
package.json
test/
underscore-min.js
underscore.js

~/Source/underscore
$ git show 8a654ec549a230dcab28e54c6e53c460cb9e190d
tree 8a654ec549a230dcab28e54c6e53c460cb9e190d

LICENSE
README
Rakefile
index.html
package.json
test/
underscore-min.js
underscore.js

~/Source/underscore
$ git log b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3

~/Source/underscore
$ git log 8a654ec549a230dcab28e54c6e53c460cb9e190d

~/Source/underscore
$ 

[jgonggrijp]

I can now reproduce the issue. Thanks for persevering, @zackschuster.

So we should probably just document the fact that this problem exists and explain how people can work around it. Perhaps a CLONING.md would be an appropriate place for such documentation.

Or, maybe we can somehow figure out which commits reference this tree. If we're very lucky, it might only be in some unimportant side branch that we can delete (or rebuild). We should probably try this first.

Help welcome.

[zackschuster]

i tried using git fast-export+git fast-import (using this SO answer) and that fixed the repo, but altered hashes. flask lists their zeroPaddedFilemode instructions under contributing, so copying their text to CONTRIBUTING.md could be an option.

[1-alex98]

Some other projects might reference this project via git in their package.json. So only documenting it might not solve everybody's case. I mean if they have existing links they might not revisit the docs...

[jgonggrijp]

@1-alex98 Do you mean a package.json entry like the following?

{
    "dependencies": {
        "underscore": "github:jashkenas/underscore#master"
    }
}

and if so, could you clarify how this may lead to the fsck error? I've never seen this happen before.

In any case, changing hashes is not really an option.

[zackschuster]

npm i underscore@"github:jashkenas/underscore#master" does not throw an fsck error for me.

[jgonggrijp]

I looked for a commit that contains the b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3 tree, using this recipe, and found it in f0427da, which unfortunately is reachable from master and dates back to Underscore's earliest days in 2010. I didn't even bother to look for 8a654ec549a230dcab28e54c6e53c460cb9e190d after that, since there is no way we are going to fix b252cd0cf660db8fea1a40ae7d3ba0e34a41ffd3.

I will accept a pull request that adds a shorter version of Flask's zero-padded file workaround instructions to the CONTRIBUTING.md. If nobody else does it, of course I will do it myself eventually.