sha1: bump sha1-asm to v0.5

[tarcieri]

cc @Byron

[tarcieri]

Tested locally on an M1 Mac Mini:

$  cargo test --features asm
   Compiling sha-1 v0.9.6 (/Users/tony/src/hashes/sha1)
    Finished test [optimized + debuginfo] target(s) in 0.81s
     Running unittests (/Users/tony/src/hashes/target/debug/deps/sha1-8de686712332341a)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/lib.rs (/Users/tony/src/hashes/target/debug/deps/lib-9426b19624108eb1)

running 2 tests
test sha1_main ... ok
test sha1_1million_a ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

   Doc-tests sha1

running 1 test
test src/lib.rs - (line 5) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s

[Byron]

Thank you sooo much, this is an absolute game changer on M1 which was previously hampered by a slow Sha1 at least compared to what git had available.

Now verifying a 1.4GB highly compressed kernel pack takes just 17s, whereas git does the same in 37s.

Program Runs

/usr/bin/time -lp ./target/release/gixp --verbose  pack-verify -s ./tests/fixtures/repos/linux.git/objects/pack/pack-3ee05b0f4e4c2cb59757c95c68e2d13c0a491289.idx
 8:05:25 SHA1 of index done 212.8MB in 0.14s (1.5GB/s)
 8:05:26  SHA1 of pack done 1.4GB in 0.89s (1.5GB/s)
 8:05:26 collecting sorted index done 7.6M entries in 1.38s (5.5M entries/s)
 8:05:28                indexing done 7.6M objects in 1.97s (3.9M objects/s)
 8:05:42               Resolving done 7.6M objects in 13.12s (579.2k objects/s)
 8:05:42                Decoding done 95.6GB in 13.12s (7.3GB/s)
objects per delta chain length
         0: 1203659
         1: 525671
         2: 508584
         3: 486484
         4: 457922
         5: 397486
         6: 356980
         7: 314955
         8: 277760
         9: 256229
        10: 229114
        11: 208084
        12: 190069
        13: 172418
        14: 158351
        15: 143646
        16: 136790
        17: 125690
        18: 117917
        19: 112052
        20: 103557
        21: 96051
        22: 90317
        23: 84294
        24: 75801
        25: 71103
        26: 65209
        27: 60033
        28: 54895
        29: 50792
        30: 47110
        31: 44741
        32: 40509
        33: 36313
        34: 32715
        35: 29587
        36: 26242
        37: 24728
        38: 22918
        39: 20701
        40: 19205
        41: 17688
        42: 16814
        43: 15222
        44: 14160
        45: 13454
        46: 12024
        47: 10825
        48: 8948
        49: 7462
        50: 7080
        ->: 7600359

averages
        delta chain length:            9;
        decompressed entry [B]:        357;
        compressed entry [B]:          175;
        decompressed object size [B]:  12580;

compression
        compressed entries size       : 1.3 GB
        decompressed entries size     : 2.7 GB
        total object size             : 95.6 GB
        pack size                     : 1.4 GB

        num trees                     : 4518550
        num blobs                     : 2133331
        num commits                   : 947819
        num tags                      : 659

        compression ratio             : 2.04
        delta compression ratio       : 71.86
        delta gain                    : 35.20
        pack overhead                 : 2.231%
real 16.81
user 89.46
sys 11.71
          2587983872  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
              223908  page reclaims
               49791  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                 316  voluntary context switches
              237096  involuntary context switches
        440116117400  instructions retired
        260960002720  cycles elapsed
          1258295488  peak memory footprint

➜  gitoxide git:(main) ✗ /usr/bin/time -lp  git -c pack.threads=4 verify-pack -v -s tests/fixtures/repos/linux.git/objects/pack/pack-3ee05b0f4e4c2cb59757c95c68e2d13c0a491289.idx
non delta: 1203659 objects
chain length = 1: 525671 objects
chain length = 2: 508584 objects
chain length = 3: 486484 objects
chain length = 4: 457922 objects
chain length = 5: 397486 objects
chain length = 6: 356980 objects
chain length = 7: 314955 objects
chain length = 8: 277760 objects
chain length = 9: 256229 objects
chain length = 10: 229114 objects
chain length = 11: 208084 objects
chain length = 12: 190069 objects
chain length = 13: 172418 objects
chain length = 14: 158351 objects
chain length = 15: 143646 objects
chain length = 16: 136790 objects
chain length = 17: 125690 objects
chain length = 18: 117917 objects
chain length = 19: 112052 objects
chain length = 20: 103557 objects
chain length = 21: 96051 objects
chain length = 22: 90317 objects
chain length = 23: 84294 objects
chain length = 24: 75801 objects
chain length = 25: 71103 objects
chain length = 26: 65209 objects
chain length = 27: 60033 objects
chain length = 28: 54895 objects
chain length = 29: 50792 objects
chain length = 30: 47110 objects
chain length = 31: 44741 objects
chain length = 32: 40509 objects
chain length = 33: 36313 objects
chain length = 34: 32715 objects
chain length = 35: 29587 objects
chain length = 36: 26242 objects
chain length = 37: 24728 objects
chain length = 38: 22918 objects
chain length = 39: 20701 objects
chain length = 40: 19205 objects
chain length = 41: 17688 objects
chain length = 42: 16814 objects
chain length = 43: 15222 objects
chain length = 44: 14160 objects
chain length = 45: 13454 objects
chain length = 46: 12024 objects
chain length = 47: 10825 objects
chain length = 48: 8948 objects
chain length = 49: 7462 objects
chain length = 50: 7080 objects
real 36.94
user 77.51
sys 8.23
           882458624  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
              153759  page reclaims
                   4  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
               32475  voluntary context switches
              652042  involuntary context switches
            14795175  instructions retired
             5040745  cycles elapsed
             1163904  peak memory footprint

[Byron]

When building on windows, I am seeing this error (here the link to the ephemeral logs):

The following warnings were emitted during compilation:
warning: cl : Command line warning D9024 : unrecognized source file type 'src/x64.S', object file assumed
Warning: warning: cl : Command line warning D9027 : source file 'src/x64.S' ignored
error: failed to run custom build command for `sha1-asm v0.5.1`
Error: failed to run custom build command for `sha1-asm v0.5.1`
Caused by:
  process didn't exit successfully: `D:\a\gitoxide\gitoxide\target\debug\build\sha1-asm-13acdb875e822f42\build-script-build` (exit code: 1)
  --- stdout
  TARGET = Some("x86_64-pc-windows-msvc")
  OPT_LEVEL = Some("0")
  HOST = Some("x86_64-pc-windows-msvc")
  CC_x86_64-pc-windows-msvc = None
  CC_x86_64_pc_windows_msvc = None
  HOST_CC = None
  CC = None
  CFLAGS_x86_64-pc-windows-msvc = None
  CFLAGS_x86_64_pc_windows_msvc = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  DEBUG = Some("true")
  running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30037\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-Z7" "-Brepro" "-W4" "-c" "-FoD:\\a\\gitoxide\\gitoxide\\target\\debug\\build\\sha1-asm-0fd8f5c0305292c2\\out\\src/x64.o" "-c" "src/x64.S"
  cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/x64.S', object file assumed
  cargo:warning=cl : Command line warning D9027 : source file 'src/x64.S' ignored
  cl : Command line warning D9021 : no action performed
  exit code: 0
  AR_x86_64-pc-windows-msvc = None
  AR_x86_64_pc_windows_msvc = None
  HOST_AR = None
  AR = None
  running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30037\\bin\\HostX64\\x64\\lib.exe" "-out:D:\\a\\gitoxide\\gitoxide\\target\\debug\\build\\sha1-asm-0fd8f5c0305292c2\\out\\libsha1.a" "-nologo" "D:\\a\\gitoxide\\gitoxide\\target\\debug\\build\\sha1-asm-0fd8f5c0305292c2\\out\\src/x64.o"
  LINK : fatal error LNK1181: cannot open input file 'D:\a\gitoxide\gitoxide\target\debug\build\sha1-asm-0fd8f5c0305292c2\out\src\x64.o'
  exit code: 1181

For now I am trying to fix it by not using asm on windows, and wonder if there is anything else I could do.

[newpavlov]

@Byron
See RustCrypto/asm-hashes#17.

[Byron]

@newpavlov Thanks a lot, I am now subscribed.

Maybe on another note: cargo seems to try to build sha1-asm even though it's deactivated on Windows as per this part of the manifest.

[target.'cfg(windows)'.dependencies]
sha-1 = { version = "0.9.1", optional = true }

[target.'cfg(not(windows))'.dependencies]
sha-1 = { version = "0.9.1", optional = true, features = ["asm"] }

Unfortunately, without a solution, windows builds break due and right now the only way this works is to not use the asm feature anywhere. It would be such a waste and I hope somebody reading this knows a workaround.

[newpavlov]

@Byron
Have you tried resolver = "2"? You also can build you project using the MinGW toolchain (assuming you don't have dependencies reliant on MSCV).

[Byron]

Fantastic, resolver = "2" did the trick! Thanks so much!

I will keep the MinGW toolchain in mind, maybe one day I can compile windows versions locally on the MacOS. Certainly would make certain issues less frustrating.

[tarcieri]

@Byron huh, that's really odd. We test on Windows targets, and don't use resolver = "2":

https://github.com/RustCrypto/hashes/blob/dd9086a/.github/workflows/sha1.yml#L99-L121

It seems like one of your other dependencies may have been activating the asm feature?

[newpavlov]

@tarcieri
I think @Byron was using MSVC toolchain, which we do not support currently as indicated in the linked issue. The CI job uses MinGW toolchain, which works fine.

[tarcieri]

Aah hmm, maybe we should test MSVC without ASM as well, or attempt to.

[Byron]

And maybe gitoxide should test against msys2 and try to disable asm only for the MSVC toolchain. That way windows users would benefit from the performance boost, and to most it probably won't matter which toolchain compiled their binary.