replace piz with rc-zip

[luizirber]

rc-zip can work on top of regular files (not memmap-ed files like piz).

It is also much easier to interact with.

Pending: more perf tests, but mem-wise I ran the command from #1909 (comment) and was seeing ~110MB max memory consumption.

---

Sadly, I think this will be draft for quite some time, it needs Rust 1.74 to compile =/
(it also broke wasm, need to do some splitting in src/core/src/storage.rs to be able to compile a subset...)

[codecov]

Codecov Report

Attention: Patch coverage is 55.00000% with 9 lines in your changes are missing coverage. Please review.

Project coverage is 86.87%. Comparing base (24ab89c) to head (37343ec).

Files	Patch %	Lines
src/core/src/storage.rs	55.00%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           latest    #3012      +/-   ##
==========================================
- Coverage   86.88%   86.87%   -0.02%     
==========================================
  Files         136      136              
  Lines       15543    15550       +7     
  Branches     2637     2637              
==========================================
+ Hits        13505    13509       +4     
- Misses       1736     1739       +3     
  Partials      302      302              

Flag	Coverage Δ
hypothesis-py	25.44% <ø> (ø)
python	92.87% <ø> (ø)
rust	61.44% <55.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bluegenes]

Sadly, I think this will be draft for quite some time, it needs Rust 1.74 to compile =/

What do we need in order to switch to 1.74? Is it just time, other packages, etc?

[luizirber]

What do we need in order to switch to 1.74? Is it just time, other packages, etc?

We don't really have any rules, but I usually check

• What is supported in debian stable? https://packages.debian.org/search?keywords=rustc
• What is supported in ubuntu? https://packages.ubuntu.com/search?keywords=rust-all&searchon=names&exact=1&suite=all&section=all
• https://repology.org/project/rust/information used to be useful, but now it is a bit messy to read.

1.74 was released Nov 16th, which is fairly recent for distro standards/release cycles. 1.74 is also a pretty big release due to projections in opaque return types, which is exactly the feature that rc-zip uses and makes async programming much easier. Not that we do async, but many, many crates are going to MSRV to 1.74 because of that...

Maybe a good rule of thumb is checking what rust is available on the places we do packaging:

• pypi - we can choose
• conda-forge - 1.76
• pyodide - nightly-2024-01-29?!?!?!?
• spack - 1.75, about to add 1.76
• nixpkgs (1.73 on latest release, 1.76 on rolling)

But these are fairly fast moving places =]

More links:

• Fairly long discussion on MSRV and linux distros: https://internals.rust-lang.org/t/rust-msrv-policy-and-linux-distros/17074
• I don't expect anyone running sourmash on embedded systems, but Ferrocene is 1.68 at the moment