Unable to clone error object not found

[steven-sheehy]

After upgrading from v0.12.2 to v0.13.1 I'm getting the below error:

{"level":"error","ts":"2021-05-28T23:30:28.660Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"hedera-mirror-node","namespace":"common","error":"unable to clone 'https://github.com/hashgraph/hedera-mirror-node.git', error: object not found"}

Here's my GitRepository:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: hedera-mirror-node
  namespace: common
spec:
  ignore: |
    /*
    !/charts
  interval: 1m0s
  url: https://github.com/hashgraph/hedera-mirror-node.git

Editing the source-controller deployment back to v0.12.2 and it works again.

[L3o-pold]

A workaround is to use gitImplementation: libgit2 from https://fluxcd.io/docs/components/source/gitrepositories/

[stefanprodan]

This is an upstream bug introduced in go-git v5.4 go-git/go-git#323

[starkers]

I hate working with go-git so much lol..

[zeripath]

Could you try go-git/go-git#326 to prove that fixes this for you?

[stefanprodan]

@zeripath even with that PR, Flux is still broken, all push actions result in empty git-upload-pack. @hiddeco opened an issue here to track this: go-git/go-git#328

Flux users, please switch to libgit2, I can understand this has caused major incidents in your production setups and I'm very sorry, it's my fault as I wrongly assumed go-git unit tests covered the changes in v5.4.0

[zeripath]

OK I've proposed reverting go-git/go-git#303 in go-git/go-git#329. I'm genuinely sorry about this - I had hoped that the testcases covered it as it was difficult to even begin to imagine writing a test case that covers this.

My suspicion is that the problem is due to other reads of the packfile causing the the reader to seek away from where it was expecting to be. It would require using ReadAt(...)s instead of Read(...) throughout the code or at least shimming the Read(...) with a thing that wrapped the ReadAt(...) with an offset. So any testcase would have to be a lot more complex - much more like the problems you're seeing.

---

Now why did I write this PR?

The reasoning I wrote the initial PR is that without this - (and libgit2 suffers this problem too) - large objects are read completely into memory. So if you have a 100MB object and want to read it - well that whole object is ending up in memory. More concerningly though, the most easy to use go-git Storage interface will cause this read to happen when you stat the object which means that even if you try to avoid loading these objects you still end up doing so.

Clearly this PR was wrong though and will require a lot more work to fix it so I think reversion is the correct step.

[hiddeco]

The reasoning I wrote the initial PR is that without this - (and libgit2 suffers this problem too) - large objects are read completely into memory. So if you have a 100MB object and want to read it - well that whole object is ending up in memory.

This was the precise reason we were (initially) very happy to see the changes land, as we have seen this too. If you end up submitting a revised version of the PR, please notify us somewhere for testing, as we have now evidence that we are a great guinea pigs :-)

Lastly, things like this happen, do not worry about it too much 🌻

[zeripath]

go-git/go-git#329 is now merged