New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cargo publish multiple packages at once #1169
Comments
This would be awesome ;) |
I suspect we may want to have a story with #883 if/when we implement this: I could easily imagine unintentionally publishing crates by not realising that I'm depending on them. |
Is there any progress on this? I think this would a very good thing to have. One of the nice things about crates is that they are modular, thus reduce the compile time of larger projects. However, if you are developing a large library compile times can get pretty slow (when it hits >2min I start to compile after 50% of the fixing, so it can compile while I'm fixing the other 50% of the issues). In this case you can split the library into sub-crates, where each sub-crate is in some sense standalone, or a build-up over "core" structure. However, each one of them would not make any sense on its own. Publishing each individually does not make any sense also and not packaging it together. If we can have this kind of publish --all this would give as an option to reduce compile times without any drawback from the user and publishing side of the library. |
Any updates whether this will be implemented and when? |
@matthiasbeyer I think this fell through the cracks when we did workspaces. This issue was supposed to track wrapping @alexcrichton what's the status? |
@wycats AFAIK this has always been in the 'nice to have' category and hasn't progressed to the 'someone has put time into designing this' category. |
One thing I just thought about: When doing a workspace release, cargo should build everything and after everything is fine, publish all crates at once... if that is even possible. Not like "build & publish the first crate and continue for each crate" but rather "build everything, then publish everything". |
It seems that in a |
I created a simple PR to deal with this annoying issue; it permits to execute
Unluckily I did not find a way to implement this logic because cargo requires that all the dependencies of a package are in the repository or the package phase (when the tarball is created) fails; consequently, before publishing a package with a "path" dependency, that dependency must be in the repository. |
Something else I would want that I don’t think that can handle is only publishing updated packages, if one of the packages already exists at its current version number it should be downloaded and verified that the new package is identical. |
being new to workspaces but not to cargo this feels very much like a paper cut. My finger just got nipped when I tried to publish a new workspace project by following the docs ...then realized that order matters. The validation of packages will fail of one of the workspace packages depends on another in the same release but which may not have been published first. In my case it's a very simple ordering but for those new to cargo, something like |
It would definitively help me with imag where I publish over 50 crates in one release! |
https://github.com/Byron/google-apis-rs and https://github.com/rusoto/rusoto would also like benefit greatly from this |
This sounds useful, but I'm wondering what the exact behaviour should be. What if you have a |
@torkleyy interesting point. It probably should not modify the Cargo.toml at all, but it would be worth considering whether or not cargo should allow for E.g. If building Simplified StepsI'm imagining
It would be worth considerinng if step 4 should be a special "atomic step" recognised by crates.io so that if for some reason the net drops out or there's a crash the user doesn't end up with only half of their packages published. |
@mitchmindtree I would like to also see an extra step between 3 and 4 doing full workspace package validation to replace the current pre-publish validation. This does the normal per-package validation steps with 2 changes:
|
I started working on this here: https://gitlab.com/torkleyy/cargo-publish-all |
I would like to say thank you to @epage and everyone who worked on this feature. It has made my life publishing crates from https://github.com/apache/arrow-rs much better |
@torkleyy I just tested your tool on my workspace I'm trying to publish on https://github.com/umccr/htsget-rs/tree/better_ci but no output is returned and fails with exit code 1. OTOH, I'm trying to publish my first set of crates ever is turning to be a fun circular ride... not the UX I had on the rest of cargo subcommands :_S Here's me trying to "go for the leaves first" and hitting some walls: /cc @mmalenic % cargo publish -p htsget-http-actix
Updating crates.io index
Packaging htsget-http-actix v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/htsget-http-actix)
error: failed to prepare local package for uploading
Caused by:
no matching package named `htsget-http-core` found
location searched: registry `crates-io`
required by package `htsget-http-actix v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/htsget-http-actix)`
% cargo publish -p htsget-http-core
Updating crates.io index
Packaging htsget-http-core v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/htsget-http-core)
Verifying htsget-http-core v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/htsget-http-core)
error: failed to verify package tarball
Caused by:
no matching package named `htsget-search` found
location searched: registry `crates-io`
required by package `htsget-http-core v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/target/package/htsget-http-core-0.1.0)`
% cargo publish -p htsget-search
Updating crates.io index
Packaging htsget-search v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/htsget-search)
Verifying htsget-search v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/htsget-search)
error: failed to verify package tarball
Caused by:
no matching package named `htsget-test-utils` found
location searched: registry `crates-io`
required by package `htsget-search v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/target/package/htsget-search-0.1.0)`
% cargo publish -p htsget-test-utils
Updating crates.io index
Packaging htsget-test-utils v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/htsget-test-utils)
Verifying htsget-test-utils v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/htsget-test-utils)
error: failed to verify package tarball
Caused by:
no matching package named `htsget-http-core` found
location searched: registry `crates-io`
required by package `htsget-test-utils v0.1.0 (/Users/rvalls/dev/umccr/htsget-rs/target/package/htsget-test-utils-0.1.0)` |
@brainstorm Looks like you have a cycle involving a dev-dependency. The simplest way of resolving that is to remove the cycle by removing the In |
Trying to summarize this thread with some of my thoughts Prior art
High-level path
This intentionally leaves of "don't publish if its already published" as I see that as separate, though related, to this issue and it has logic/policies to be worked out. Multi-package packagingRequirements
For dry-run, we'd need to build in-order and patch in the .crates that we already packaged (see #1169 (comment)). When verifying, we likely should detect dependency cycles to give people errors early (#1169 (comment)). Multi-package publishingBecause this isn't atomic, we should try to do all verification upfront so there aren't errors along the way
When publishing, we'll also have to do it in-order. If a publish PUT fails, we need to be clear about what didn't get published for people to recover. We should track the wait-for-publish timeouts from #11602 on a per-package basis so
Additional InfoFor me, the biggest open question is how to build the DAG for packaging and then pass that up to publishing. Last I looked, the main ways I saw for interacting with the DAG was compilation which is too heavy handed for what we need. For me, the biggest area of complexity is the dependency + timeout tracking while publishing in-order. |
Dependency cycles are actually allowed, as long as the package names have been previously registered on crates.io without cycles. There's no version checking performed by crates.io, just a simple name check, so you can publish crates that depend on future unpublished versions of other crates. |
I've edited it to clarify that is for when verifying as that does not support cycles (minus the automatically removed dev-dependencies) |
An interesting challenge for us to keep in mind with this is registries that rate-limit, both in terms of finding the right strategy for backing off but also the right UX so someone doesn't publish 300 crates and it takes 24 hours without any clear indication. |
This issue is |
It would be nice to have a flag to
cargo publish
which publishes all local packages in a DAG fashion.Notes (edit ehuss):
--dry-run
should work correctly (will need to pretend that the previous crates have been published, maybe viapatch
?) See also Add a dry-run publish API crates.io#1515.The text was updated successfully, but these errors were encountered: