Object access performance #62
Replies: 7 comments 3 replies
-
|
Note that the compile times are pretty high at least on the first build due to tinyvec triggering a weak spot in polonius right now. |
Beta Was this translation helpful? Give feedback.
-
|
I also added a test with parallel access to the object database, which does provide some speedup indeed. Here on And here on Rust. (Note that one core was compiling while the benchmark above was running) |
Beta Was this translation helpful? Give feedback.
-
|
For fairness, the parallel version of libgit2 was also added, here is the result for the gitoxide repo: And here is the one for the Rust repository There are two observations:
|
Beta Was this translation helpful? Give feedback.
-
|
For the fun of it, in order to get the object size in the fastest possible way, I applied this patch: Now the results look different as libgit2 is able to safe a lot of time when handling packed objects - it will just have to decode the result size of the last delta in the chain which happens to be the object of interest. gitoxide has no such code path and won't have it unless there is an actual application, but…here are the values. Most notably, libgit2 now looses even more performance in the multi-threaded case which might indicate some globally locked state and contention around it. |
Beta Was this translation helpful? Give feedback.
-
|
The original benchmark was using the summed lengths of the full objects as a proxy for confirming that it had fully reconstructed the data and not elided any of that work. It's absolutely possible to get the object sizes without reconstructing the objects themselves, but that'd be an entirely different benchmark. (It was still worth testing, though.) |
Beta Was this translation helpful? Give feedback.
-
|
In the most recent iteration and support for bigger caches, performance changed once more and to the better. After all, to get faster one has to parallelize efficiently and avoid work through maximizing cache hits. With the With the Rust repository it looks a little differently, with Especially the parallel performance with a well-configured cache will allow for much smaller losses when building packs due to decoding objects for diffing, allowing more time to spend on what matters there most. Regarding the build failures do you think this is because of V7 instrinsics being used/tested on a CPU that doesn't have them? I remember having generalized some compile time includes to get other tests to work, maybe that backfires in this way now (?). |
Beta Was this translation helpful? Give feedback.
-
|
There is a new Out of curiosity, I added a 5x larger cache to see what happens when dealing with bigger repositories. And here is 20x for single-threaded performance only - greater caches bring neglible results for access patterns we see during traversal. I think the best possible performance is only achieved with the traversal type (and index) performed when building an index from a pack file to never decode an entry twice, and bigger caches here would certainly never even get close to that. |
Beta Was this translation helpful? Give feedback.
-
|
While working towards all the infrastructure for pack generation there is now an iterator over all object ids in a git database. This is (sans the 'only reachable' constraint) pretty much what we hoped to get as input for the So here we go, with the reading of For the gitoxide repository… For a snapshot of the Rust repository… It's interesting to see how caches (at least of that size) now don't help at all anymore to speed up pack decoding, slowing down overall performance (glad we have static dispatch/monomorphism here).
It gets more interesting with multiple cores, where I find this outcome very insightful and am very optimistic that the very fine-grained and customizable caching concept used in Repeated access of the same objects, for instance, as might happen when traversing multiple trees which are mostly similar, is certainly best sped up with an object traversal cache, which is only used during traversal and effectively sits on top of the pack access cache. With that in place, I believe typical tree traversals will be very fast while providing perfect control over memory consumption. |
Beta Was this translation helpful? Give feedback.
-
|
Bonus: the highly compressed linux kernel pack
|
Beta Was this translation helpful? Give feedback.
-
|
For fun I ran What's interesting here is that As Edit: Here is the same repository after
This probably means that the |
Beta Was this translation helpful? Give feedback.
-
As of now, object access via
git_odb::compound::Dbis absolutely naively implement without any lookup cache. This discussion holds measurement results and will track efforts to eventually make it faster.How to run it:
The above yields…
…which isn't bad at all at 0.7x performance!
It looks a bit different if the pack is less perfect though, like running it directly on this repository…
…which only reaches 0.3x performance.
Causes for slow speed
object-accessprogram is configured to use nightly and polonius to avoid this penalty though.pack::cache::noopand thus incurs high overhead when encountering long delta chainsInstrumentation
A lot of zlib work, more than needed due to low caching.
In comparison, git2…
Maybe notable: gitoxide spends twice as much time in memmoves for some reason.
Beta Was this translation helpful? Give feedback.
All reactions