Tracking Issue for serving an index over HTTP

[ehuss]

Summary

RFC: #2789
Implementation: #10470
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#sparse-registry
Issues: A-sparse-registry Area: http sparse registries

This is a tracking issue for RFC #2789, an experimental extension to serve the index over HTTP instead of via git.

Unresolved issues

• Cache invalidation
  • Currently 1 minute TTL. Too long for publish, too short for effective caching.
  • Minor concern about how long an invalidation takes (Cloudfront says 10 to 100 seconds)
  • The cost is expected to negligible.
  • Send invalidations for index.crates.io when updating crates crates.io#4913
• Make sparse+ and registry+ more consistent. HTTP registry implementation #10470 (comment) (resolved Add new SourceKind::SparseRegistry to differentiate sparse registries #11209)
• Find a better name (and structure) for index_version HTTP registry implementation #10470 (comment)
• Handle user urls that start with http. They currently imply registry+. But if sparse+ are more common then it would be good to auto detect. RFC: Serve crates-io registry over HTTP as static files rfcs#2789 (comment)
• Have a plan for lockfiles.
• Implement support in crates.io Upload index metadata to index/ when publishing new crates crates.io#4661
• Check that crates.io is willing to maintain their current system at production levels of traffic.
• Teach cargo that crates.io has two equivalent indexes. Teach cargo that crates.io has two equivalent indexes. #10722
• Make sure we are future compatible with a merkle tree design. Sparse registry indexes should be viewable atomically #10928
• The RFC is intentionally sparse on details, it is almost an eRFC. Do we need another RFC/PR to discuss the details we have chosen?
• Does not support fuzzy queries. Is that a blocker? Only used by error messages. This was a misunderstanding, indexes don't support name-based typo suggestions at all. Sparse supports the same case-folding and -/_ fuzzy comparison as git indexes. Sparse fundamentally won't be able to support typo suggestions, but since cargo hasn't handled that in the past, this isn't a regression, and more like a "can't do that in the future" thing.
• Investigate publish issues. 1 minute update is quite slow if publishing multiple packages (and cargo does not yet have retry logic).

Current best way to test

To try it out, add the -Z sparse-registry flag on nightly-2022-06-20 or newer build
of Cargo. For example, to update dependencies:

rustup update nightly
cargo +nightly -Z sparse-registry update

The feature can also be enabled by setting the environment variable
CARGO_UNSTABLE_SPARSE_REGISTRY=true. Setting this variable will have no effect on stable
Cargo, making it easy to opt-in for CI jobs.

If you see any issues please report them in new tickets here in the Cargo repo. The output of Cargo
with the environment variable CARGO_LOG=cargo::sources::registry::http_remote=trace set
will be helpful in debugging.

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Implementation history

• Registry functions return Poll to enable parallel fetching of index data #10064

• HTTP registry implementation #10470

• Refactor RegistryData::load to handle management of the index cache #10482

• Require http-registry URLs to end with a '/' #10698

• Make -Z http-registry use index.crates.io when accessing crates-io #10725

• Improve testing framework for http registries #10738

• Make is_yanked return Poll<> #10830

• Fix publishing to crates.io with -Z sparse-registry #10831

• Refactor check_yanked to avoid some duplication #10835

• Change progress indicator for sparse registries #11068

• Add retry support to sparse registries #11069

• Fix sparse registry lockfile urls containing 'registry+sparse+' #11177

• Add new SourceKind::SparseRegistry to differentiate sparse registries #11209

• Add configuration option for controlling crates.io protocol #11215

• Upload index metadata to index/ when publishing new crates crates.io#4661

• Drop index/ prefix when uploading crate metadata. crates.io#4826

[mightyiam]

Hey, would this mean that when cargo publish exits with 0 the published crate version is actually available (which is not the case now, due to CDN)?

[kornelski]

It doesn't address this problem. There is still going to be a delay between cargo publish and the crate being available globally. It would be nice to improve this, but it's more of an issue for server-side crates-io implementation than the registry protocol.

[epage]

@mightyiam #9507 is the issue you want to be following.

For background, publishing a crate used to be a blocking operation but to speed up crates.io, they made cargo publish just put the crate into a queue for later publishing, making the operation asynchronous.

Tools like cargo-release poll the server. We could easily do similar in cargo until an improved registry protocol is made. I was looking at implementing it but there was a large hurdle for writing the relevant tests. The new sparse registry code has improved the test infrastructure so I could now write the relevant tests.

[tigregalis]

Can the unstable feature be enabled via .cargo/config.toml?

[Eh2406]

Yes, I think it is:

[unstable]
sparse-registry = true

[ehuss]

Just an update for anyone following this issue. A proposal for how to configure git vs http is available at https://hackmd.io/@rust-cargo-team/B13O52Zko. Additionally #10964 and #10965 contain some more background.

[luciusmagn]

Hi, has there been any progress on this? Anything I can help with? We would really like to help this get into stable ASAP

[weihanglo]

Thank you for being interested in this feature! The Cargo team and contributors are currently working hard on this topic, as well as collaborating with crates.io team to build up the infrastructure needed (rust-lang/crates.io#5200, rust-lang/crates.io#5112, rust-lang/crates.io#5066, etc).

One of the things we're happy to collect is feedback for this feature. Personally I'd appreciate any user feedback, especially about user experiences. You can find more relevant issues from label https://github.com/rust-lang/cargo/labels/Z-sparse-registry, or in the description of this issue, which contains the development history and unresolved issues.

[jasal82]

Any estimation on when this will be available on stable? Using the sparse registry feature is the only available solution for the following issue in Artifactory https://www.jfrog.com/jira/browse/RTFACT-27248 which is currently a blocker for us.

[kornelski]

@jasal82 for corporate proxies, use net.git-fetch-with-cli = true.

[jasal82]

@kornelski We're already using that option in the clients. However, for production use we must not use the upstream repositories but Artifactory remote caches instead. Now the problem is that Artifactory cannot properly update its remote caches from the upstream repo when a new previously uncached package is requested. This is definitely a bug in Artifactory but they refuse to fix it and the only answer we got from them is that we should use sparse registries instead which seems to work.

[ahicks92]

I'd also be interested on when this might hit stable. We also have CI difficulties/runarounds in docker because Cargo deciding that it needs to clone the index from scratch takes a long time for whatever reason, and I suspect we're not alone in that.

For the curious our docker solution is cargo install lazy-static which forces cargo to get an index before failing out because there's no binary target.

[lnicola]

For what it's worth, rust-analyzer also tends to get blamed for this ("but after I closed Code, I ran cargo clean and cargo check and they were fast, it's RA that's slow!") because it calls cargo metadata when loading the project.

[ahicks92]

It's bad even on my gigabit internet at this point if I'm being honest. But since that's once per install that doesn't seem worth complaining about, or something like that. I'll be honest though: I dread that progress bar nowadays.

[ehuss]

A proposal to stabilize this has been posted at #11224.

[ehuss]

Sparse registries are now stabilized and will be available in Rust 1.68, which will be released on 2023-03-09. I'm going to close this tracking issue as I think it has served its purpose. There is still some future work to be continued, such as better support for third-party registries. Those issues can be tracked in the A-sparse-registry Area: http sparse registries label.

[Christoph-AK]

I'm super glad this got merged, but I am having trouble finding the necessary config steps to enable the new behaviour.
Most of the documentation, like in the first post in this issue, still mention it as unstable feature to be enabled with -Z.
Adding [registries.crates-io] protocol = sparse to my cargo.toml doesn't work, i suppose it needs to be added to some cargo config file somewhere else.
Could someone that got it running write some annotations or documentation in some easy to find places?

[JockeTF]

Cargo.toml doesn't work

Agreed, this is a great feature! According to the release notes you're supposed to add the setting to .cargo/config.toml, not Cargo.toml. Either that, or set the environment variable CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse. The release notes have some links to more information as well!

[jose-mut-lopez]

you're supposed to add the setting to .cargo/config.toml, not Cargo.toml.

thanks for this and those links, but for future outsider readers like me, it looks like this became the default in cargo 1.70 which was released on June 2023 (https://blog.rust-lang.org/2023/06/01/Rust-1.70.0.html#sparse-by-default-for-cratesio), so a better approach now IMO is just to rustup update, which will use the fast index on all repos by default without any per-repo config.