Add local registry overlays #13926

jneem · 2024-05-17T21:14:41Z

This PR adds (private to cargo internals) support for local registry overlays, in which you can locally pretend to add packages to remote registries; the local packages will have the same source ids as the remote registry that you're overlaying.

There are two ways to set up these overlays: programmatically using GlobalContext::local_overlays and through the __CARGO_TEST_PACKAGE_CONFUSION_VULNERABILITY_DO_NOT_USE_THIS environment variable. You can't set up these overlays with .cargo/config.

The motivation for this is packaging workspaces. When we're packing a workspace, we'd like to be able to pretend (for lockfile generation and verification) that some workspace packages are already published even though they aren't.

rustbot · 2024-05-17T21:14:46Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

Eh2406 · 2024-05-20T13:57:15Z

crates/cargo-test-support/src/lib.rs

+        if let Some(ref mut p) = self.process_builder {
+            let env_value = format!("{}={}", url, path);
+            p.env(
+                "__CARGO_TEST_PACKAGE_CONFUSION_VULNERABILITY_DO_NOT_USE_THIS",


Nit: The term of art for this kind of vulnerability is "Dependency Confusion". As with so many names in computer science, it's better to have consistency then trying to find a better name. This term was coined in the blog post https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610 which might be worth linking in comments about how users could easily miss use this.

src/cargo/sources/overlay.rs

Eh2406 · 2024-05-20T14:06:41Z

Catching Eric up on some of the context, Arlo and I discussed the questions from #10948 (comment) at office hours on Thursday. Adding this API, exclusively for internal use, seemed like the least hacky solution.

Cargo.toml

src/cargo/sources/overlay.rs

src/cargo/util/context/mod.rs

torhovland · 2024-05-21T08:42:24Z

src/cargo/sources/overlay.rs

+/// expose this publicly. However, it is useful for some very specific private
+/// things, like locally verifying a bunch of packages at a time before any of
+/// them have been published.
+pub struct OverlayConfusionAttack<'gctx> {


Frankly, I think OverlaySource is a better fitting name 🙂

That name definitely makes more sense, yes. But in office hours someone suggested giving it a scary-looking name to make people less likely to misuse it...

I like OverlaySource being the suffix. Maybe DependencyConfusionThreatOverlaySource?

Co-authored-by: Tor Hovland <55164+torhovland@users.noreply.github.com>

jneem · 2024-05-23T16:00:14Z

Thanks for the review! I think this is ready for another look -- we managed to successfully use this for #10948 as discussed.

epage · 2024-05-29T18:33:27Z

tests/testsuite/registry_overlay.rs

+    // An older version in the local registry doesn't confuse it.
+    Package::new("baz", "0.0.1")
+        .alternative(true)
+        .local(true)
+        .publish();


I don't think this is really testing anything because baz is too low of a version for the version requirement

The motivation here was to test that the overlay is correctly the union of the versions in the overlay and the overlaid registries. (If it were only returning the overlay versions, it would only return the too-old version and fail to resolve)

Except this is relying on the user to look at the error and know the fine distinction of whether it says baz@0.0.1 exists

We also should probably have a test the other direction, that the overlay source properly shadows the regular source.

Got it. I'm now testing both directions, and checking the appropriate versions in the stderr.

tests/testsuite/registry_overlay.rs

epage · 2024-05-29T18:35:57Z

tests/testsuite/registry_overlay.rs

+    Package::new("baz", "0.1.1").dep("baz", "=0.0.1").publish();
+    Package::new("baz", "0.0.1")
+        .alternative(true)
+        .local(true)
+        .publish();


Should we have the local / selected version depend on the registry version? Thats more like what we'll be dealing with for cargo package

I don't think so, but also happy to discuss it further. My understanding was that the behavior of cargo package w.r.t. intra-workspace deps should be exactly the same as if the deps were published before packaging. Resolution should have the option to select the versions that are being packaged simultaneously, but it's also allowed not to.

Yes, that should be the behavior but we aren't testing that situation to confirm it. cargo package will involve a local package depending on an overlayed package which may depend on a registry package.

I guess there are also situations where you depend on a registry package which will depend on the overlaid package (if clap_complete used criterion it would both depend on clap directly and through criterion). So maybe its best to cover both cases.

Ah, makes sense, I just misunderstood your original comment. I'm now testing both directions.

I guess there are also situations where you depend on a registry package which will depend on the overlaid package

I think the registry package will never require an overlaid package, right? Because the registry package can satisfy all its dependencies from the registry. I added a test where the registry package will pick up an overlaid dependency because it's newer (but compatible with) the registry version of that dependency.

tests/testsuite/registry_overlay.rs

epage · 2024-05-29T18:41:09Z

src/cargo/util/context/mod.rs

+    /// affect any newly created `PackageRegistry`s.
+    ///
+    /// See [`crate::sources::overlay::OverlayConfusionAttack`] for why you shouldn't use this.
+    pub fn local_overlays(&self) -> RefMut<'_, HashMap<SourceId, PathBuf>> {


I'd hope we can avoid putting this on GlobalContext but its hard to say without reviewing the follow up at the same time. Should we defer this to the follow up PR?

My motivation is that it gets messy adding and removing from the Global Context and if we can directly instantiate things, the easier it will be to maintain.

I can give a little more context before we decide whether or not to defer. cargo package needs the overlay to take effect during a call to ops::compile. AFAICT there are three possibilities:

put the overlay configuration in GlobalContext

put the overlay configuration in the ephemeral workspace that cargo package creates

have cargo package do something lower-level than ops::compile

The workspace option has the advantage that the overlay configuration doesn't need to be removed (because the workspace is discarded). It seems like it would just require a little work in resolve_ws and resolve_ws_with_opts to get the overlay configuration out of the workspace and insert it into the PackageRegistry

The workspace seems the most beneficial. Whichever way we go, still seems like it could be split out into a follow up PR

epage · 2024-05-29T18:47:17Z

src/cargo/sources/config.rs

+            let src = id.load(self.gctx, yanked_whitelist)?;
+            return self.overlay_source(id, src, yanked_whitelist);


We only call overlay_source on the result of a load

Should we name it fn load and perform the load and apply the overlay? I worry that the current structure could make it easy to overlook applying the overlay while this pattern would make it the default to go through Self::load and we'd be less likely to introduce bugs to this

Unfortunately we're already in a fn load. I renamed overlay_source to load_and_overlay, and modified it to do the load and the overlay. I think this is an improvement but not sure it it really addresses the concern.

epage · 2024-05-29T18:48:32Z

src/cargo/sources/config.rs

+        for (key, path) in gctx.local_overlays().iter() {
+            let overlay_id = SourceId::for_local_registry(path)?;
+            base.overlays.insert(*key, overlay_id);
+        }
+
+        if let Ok(overlay) =
+            gctx.get_env("__CARGO_TEST_DEPENDENCY_CONFUSION_VULNERABILITY_DO_NOT_USE_THIS")
+        {
+            let (url, path) = overlay.split_once('=').ok_or(anyhow::anyhow!(
+                "invalid overlay format. I won't tell you why; you shouldn't be using it anyway"
+            ))?;
+            let overlay_id = SourceId::for_local_registry(path.as_ref())?;
+            base.overlays.insert(SourceId::from_url(url)?, overlay_id);
+        }


Should we move this hack up into the caller? PackageRegistry is the immediate caller but maybe a layer above that? That would help with the GlobalContext. The main downside is it would make it more annoying to support __CARGO_TEST_DEPENDENCY_CONFUSION_VULNERABILITY_DO_NOT_USE_THIS

Thoughts?

I think if we moved the overlay configuration into Workspace as discussed below then this could be pushed up into resolve_ws and resolve_ws_with_opts

epage · 2024-05-30T20:12:40Z

tests/testsuite/registry_overlay.rs

+    )
+}
+
+fn simple_project() -> Project {


Let's avoid using this. This reduces "duplication" but that is false code duplication. They have the appearance of being the same but they do not share requirements. Instead it affects the code locality (the ability to read tightly coupled code). I'm trying to double check what these tests are doing and jumping back and forth is making this more difficult.

Add local registry overlays

e136e6c

rustbot assigned ehuss May 17, 2024

rustbot added A-configuration Area: cargo config files and env vars A-testing-cargo-itself Area: cargo's tests S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 17, 2024

jneem added 2 commits May 17, 2024 17:26

Downgrade libc to avoid the yanked version

89c2b3e

Maybe make the paths work on windows

88c9aac

Eh2406 reviewed May 20, 2024

View reviewed changes

src/cargo/sources/overlay.rs Outdated Show resolved Hide resolved

Eh2406 reviewed May 20, 2024

View reviewed changes

src/cargo/sources/overlay.rs Outdated Show resolved Hide resolved

Eh2406 reviewed May 20, 2024

View reviewed changes

src/cargo/sources/overlay.rs Outdated Show resolved Hide resolved

Review comments

3279632

torhovland reviewed May 21, 2024

View reviewed changes

jneem and others added 2 commits May 21, 2024 13:16

Typo

ce7670a

Co-authored-by: Tor Hovland <55164+torhovland@users.noreply.github.com>

Update comment

bcc3d73

jneem mentioned this pull request May 22, 2024

[WIP] Package workspaces #13947

Draft

4 tasks

jneem added 3 commits May 21, 2024 20:17

Bump libc back up

9d7e8f4

Also overlay named alternate registries

5a87535

Add a test

e999fe7

jneem requested a review from Eh2406 May 23, 2024 15:54

epage reviewed May 29, 2024

View reviewed changes

tests/testsuite/registry_overlay.rs Outdated Show resolved Hide resolved

epage reviewed May 29, 2024

View reviewed changes

tests/testsuite/registry_overlay.rs Outdated Show resolved Hide resolved

epage reviewed May 29, 2024

View reviewed changes

jneem added 3 commits May 29, 2024 16:57

Rename and refactor

0bb063a

Test simplifications

3b2e255

Fix doc link

986e5a0

epage reviewed May 30, 2024

View reviewed changes

jneem added 2 commits May 30, 2024 15:29

Keep the project def with each test

9908b41

Test improvements

12fc221

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add local registry overlays #13926

Add local registry overlays #13926

jneem commented May 17, 2024

rustbot commented May 17, 2024

Eh2406 May 20, 2024

Eh2406 commented May 20, 2024

torhovland May 21, 2024

jneem May 21, 2024

epage May 29, 2024

jneem commented May 23, 2024

epage May 29, 2024

jneem May 29, 2024

epage May 30, 2024 •

edited

jneem May 30, 2024

epage May 29, 2024

jneem May 29, 2024

epage May 30, 2024

jneem May 30, 2024 •

edited

epage May 29, 2024

jneem May 29, 2024

epage May 30, 2024

epage May 29, 2024

jneem May 29, 2024

epage May 29, 2024

jneem May 29, 2024

epage May 30, 2024

		let src = id.load(self.gctx, yanked_whitelist)?;
		return self.overlay_source(id, src, yanked_whitelist);

Add local registry overlays #13926

Are you sure you want to change the base?

Add local registry overlays #13926

Conversation

jneem commented May 17, 2024

rustbot commented May 17, 2024

Choose a reason for hiding this comment

Eh2406 commented May 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jneem commented May 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epage May 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jneem May 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epage May 30, 2024 •

edited

jneem May 30, 2024 •

edited