Should we use crc32fast?

[edward-shen]

The readme states,

Probably at no cost one could upgrade to the crc32fast crate, but it looks unmaintained and builds more slowly.

I think rather than unmaintained, it's just a mature project—open PRs are really more QoL for the maintainers than the code quality, and for what it's worth the API looks clean to not need new updates. To add on, it looks like it's still (relatively) maintained: there was a commit on 2020-10-21.

As for build speed, is it significant?

I ask because it looks like crc32 is pretty ubiquitous in git-odb, which could mean a pretty significant improvement in perf.

[Byron]

Thanks for bringing this up. Initially I thought I already use crc32fast somewhere, but was wrong. It also turns out that crc32fast is already coming in with the flate2 crate.

➜  gitoxide git:(main) cargo tree -i crc32fast
crc32fast v1.2.1
└── flate2 v1.0.20
    └── git-odb v0.15.0 (/Users/byron/dev/github.com/Byron/gitoxide/git-odb)
        └── gitoxide-core v0.9.0 (/Users/byron/dev/github.com/Byron/gitoxide/gitoxide-core)
            └── gitoxide v0.7.0 (/Users/byron/dev/github.com/Byron/gitoxide)

On my system, a release build of crc32fast costs 0.93s whereas the one for crc costs 0.42s.

Also I can only agree that the assessment of it being unmaintained is wrong - it's a mature project without major changes.

In the light of this, I will switch over to crc32fast, and admittedly flate2 using it is a major factor.

[edward-shen]

Ya beat me to it! Have you noticed any noteworthy improvements?

[Byron]

I didn't check actually :D - maybe this is something you could check. Building an index from a pack should heat up that codepath.

Edit: It's about 2s faster, but that measurement is definitely not statistically correct.

[edward-shen]

Did a quick informal test:

• Generated a pack file from target/release/gixp pack-create d067cc46e93f3f343b1244908da338132ce4f6cd -r . > tmp
• Ran 100 instances of pack-index-from-data:
• On b6b58560:

time (for i in $(seq 100); do ./gixp-b6b58560 pack-index-from-data < tmp > /dev/null; done)
( for i in $(seq 100); do; ./gixp-b6b58560 pack-index-from-data < tmp > ; done  58.85s user 5.62s system 202% cpu 31.769 total

• On 11955f95 (current main as of writing this):

time (for i in $(seq 100); do ./gixp-11955f95 pack-index-from-data < tmp > /dev/null; done)
( for i in $(seq 100); do; ./gixp-11955f95 pack-index-from-data < tmp > ; done  51.14s user 5.34s system 229% cpu 24.590 total

33% 21% wall time decrease from ~32s to ~25s!

Edit: I can't math.

[Byron]

Me neither, but hyperfine can - the faster invocation is the latest version.

➜  gitoxide git:(main) ✗ hyperfine './target/release/gixp pack-index-from-data < tmp > /dev/null' 'gixp pack-index-from-data < tmp > /dev/null'
Benchmark #1: ./target/release/gixp pack-index-from-data < tmp > /dev/null
  Time (mean ± σ):     263.6 ms ±   4.3 ms    [User: 545.1 ms, System: 42.6 ms]
  Range (min … max):   259.7 ms … 272.9 ms    10 runs

Benchmark #2: gixp pack-index-from-data < tmp > /dev/null
  Time (mean ± σ):     336.0 ms ±   4.4 ms    [User: 617.6 ms, System: 42.6 ms]
  Range (min … max):   332.6 ms … 346.1 ms    10 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  './target/release/gixp pack-index-from-data < tmp > /dev/null' ran
    1.27 ± 0.03 times faster than 'gixp pack-index-from-data < tmp > /dev/null'
➜  gitoxide git:(main) ✗ hyperfine './target/release/gixp pack-index-from-data < tmp > /dev/null' 'gixp pack-index-from-data < tmp > /dev/null'

[Byron]

I think with this change, at least on intel, pack indexing should be significantly faster now than git itself. Let's see…