plumbing: format/packfile, prevent large objects from being read into memory completely

[zeripath]

This PR adds code to prevent large objects from being read into memory from packfiles or the filesystem.

Objects greater than 1Mb are now no longer directly stored in the cache
or read completely into memory.

Signed-off-by: Andrew Thornton art27@cantab.net

[jfontan]

@zeripath Thanks for the PR. I like the changes but I'm a little worried of using a goroutine for the reader. Do you think it can be done without using it?

[zeripath]

So before you merge this, there are a couple of questions that we should think about:

1. Should we really be caching these incompletely read objects? - Could the object files be changed under our feet by a repack etc? Could you end up with a broken object that it would not be easy to get a gogit repository to clean out from the cache? I don't know go-git's architecture completely so I'm not certain if that's possible.
2. Is the 1Mb limit too low for the delta'd objects? It feels appropriate for the raw object db cases as those are simple file reads - whereas a delta'd object is a little more involved and perhaps the limit should be raised to account for the complexity?

Many thanks for your work on this

[zeripath]

Sorry I've taken so long to answer my own questions here.

1. Should we really be caching these incompletely read objects? - Could the object files be changed under our feet by a repack etc? Could you end up with a broken object that it would not be easy to get a gogit repository to clean out from the cache? I don't know go-git's architecture completely so I'm not certain if that's possible.

This is fine - it's only when you get the reader that you actually do the reading.

2. Is the 1Mb limit too low for the delta'd objects? It feels appropriate for the raw object db cases as those are simple file reads

• whereas a delta'd object is a little more involved and perhaps the limit should be raised to account for the complexity?

This is also probably fine.