Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feedback / Experience / Adventures meta-issue #19

Open
ryankurte opened this issue Dec 31, 2020 · 39 comments
Open

Feedback / Experience / Adventures meta-issue #19

ryankurte opened this issue Dec 31, 2020 · 39 comments

Comments

@ryankurte
Copy link
Collaborator

A meta-issue for the experience of using this, if you have any thoughts / feelings / ideas please comment here!

@ryankurte ryankurte pinned this issue Dec 31, 2020
@fasterthanlime
Copy link

fasterthanlime commented Sep 6, 2021

Discovered cargo-binstall through cargo-watch's readme, instantly worked, no complaints.

I've been trying to run cargo binstall some-random-package, and it's often failed due to some-random-package not having the right "releases layout" / not having binaries at all / not having any cargo-binstall config. I'm wondering if cargo-binstall should fall back to building from source in these cases (maybe after asking / only if a CLI flag is passed?). I'd love for cargo-binstall to be my one-stop-shop for installing Rust utilities. If there's a binary, yay! If not, I'll take the compile time hit.

@ryankurte
Copy link
Collaborator Author

Discovered cargo-binstall through cargo-watch's readme, instantly worked, no complaints.

glad to hear it ^_^

I've been trying to run cargo binstall some-random-package, and it's often failed due to some-random-package not having the right "releases layout" / not having binaries at all / not having any cargo-binstall config.

ahh, yeah, i'm definitely open to improvements if you have ideas! have contemplated making the discovery process a bit more robust/overzealous if there's no config, or some kind of unofficial registry so where folks have CI builds we could link to their binaries without Cargo.toml config, but i think actually building them would move this from (hopefully) helpful tool to accidentally a package manager.

I'm wondering if cargo-binstall should fall back to building from source in these cases (maybe after asking / only if a CLI flag is passed?). I'd love for cargo-binstall to be my one-stop-shop for installing Rust utilities. If there's a binary, yay! If not, I'll take the compile time hit.

sounds like a great idea to me! given it's already interactive a prompt w/ ci flag to automagically do it would probably make sense?

@fasterthanlime
Copy link

given it's already interactive a prompt w/ ci flag to automagically do it would probably make sense?

That sounds great! I vaguely remembered seeing some interactive prompts already in cargo-binstall hence my suggestion.

@pinage404
Copy link
Contributor

pinage404 commented Oct 17, 2021

@pinage404
Copy link
Contributor

It's hard to understand why format :

Does the format refer to the same concept ?

@ryankurte
Copy link
Collaborator Author

thanks for the reports!

wrt. bin-dir and bin-path, this appears to be a typo / should all be called bin-dir

wrt. format, those are actually two (unfortunately named) different concepts, the first refers to the archive format (eg. .tgz), the second to the binary format (which needs a .exe appended for windows).

it would be nice to rename these to highlight the difference, and it might make sense to include the . in both cases, but i would aim to avoid any breaking changes. it's probably worth looking at whether there's a compatible way to mitigate these and definitely able to improve the documentation to clarify both!

@pinage404
Copy link
Contributor

for the format, this could be done without breaking changes, supporting both the deprecated format and the new archive-format and binary-format

passcod added a commit that referenced this issue Feb 16, 2022
This from feedback in #19:

> wrt. bin-dir and bin-path, this appears to be a typo / should all be called bin-dir

This is only a readme fix afaict, I changed all occurences of `bin-path` in there to `bin-dir`.

> wrt. format, those are actually two (unfortunately named) different concepts, the first
refers to the archive format (eg. .tgz), the second to the binary format (which needs a .exe
appended for windows).

This introduces two new substitutions:
- `binary-ext` is the old "`format` in `bin-dir`"
- `archive-format` is the old "`format` in `pkg-url`"

Contents are unchanged: `binary-ext` includes the dot, `archive-format` doesn't. That
makes it easy to upgrade and also personally I slightly prefer it that way.

The old contextual `format` is still available, "soft deprecated": it will be accepted silently
so everything will work, but all documentation will use the new syntax. In the future we
could move to a "hard deprecated" model where installing a package that uses `format`
will warn the user / tell them to report that to the maintainer. I don't think we'll ever really
be able to remove it but that should be good enough.

A cool new feature is that `binary-ext` is now usable in `pkg-url`, which will be useful for raw binary downloads:

```toml
pkg_url = "{ repo }/releases/download/v{ version }/{ name }-v{ version }-{ target }{ binary-ext }"
```

I've also added a bunch of tests to GhCrateMeta around the templating for `pkg-url`.
@auronandace
Copy link

Would it be possible to add a scan feature to see what you have installed via cargo install and list whether they are also able to be installed via cargo binstall? Sort of a binstall version of this:

https://github.com/nabijaczleweli/cargo-update

@somehowchris
Copy link
Contributor

somehowchris commented Apr 19, 2022

Something like "install from Cargo.toml" would be nice. We have a case where some people joined the team and didn't know the tools and just listing them one by one can be a long process on multiple machines.

Also this would give us the oportinity to profit from tooling like dependabot, renovate, synk,... which would give you a heads up if some tools have been updated, so you should update your tooling too (or the Cargo.toml which you then can just i.e. cargo binstall --manifest .). This could be included in things like git hooks or make files so all devs know they are running the same thing.

Also, tried to add cicd with cross and a lot more to just have binstall support. I think this is a bit hurdle for some which then just get frustrated. But I'm trying to get a github action running for something like this. (would also be lovely to have #1 with that 😄 )

But in general, just awesome, saves a lot of time

@passcod
Copy link
Member

passcod commented Apr 19, 2022

Oh, that's interesting! Are you looking for a list of crates or also something like having version requirements or a lockfile?

@somehowchris
Copy link
Contributor

somehowchris commented Apr 19, 2022

Yes, well would be nice.

Just has been a small pain point for me today. Me personally, I have a small repo with a Cargo.toml which is watched by renovate, so I know when a tool has been updated. For a) update the container image and b) actualy know what changed.

So lets say if v0.1.1 was released and I have v0.1.0 on my laptop, it would be lovely to just have something like cargo binstall update or as said cargo binstall --manifest . to update the binaries. (or as mentioned have them as githooks for something like git pull)

Edit: Also just had a thought about rust equivalent tooling to like npms husky, and tooling used alongside cargo like cargo-watch or wasm-bindgen-cli, specified in the Cargo.toml under something like package.metadata.binstall-tools to not get confused with other dependencies and to have them versioned per project as they are needed for it. This would not be included in usual tooling like renovate or dependabot but I know renovate can easily be configured to include these and they would be for sure be open to have a PR to include this (behind the scenes it would use cargos registries & functionality as usual)

@somehowchris
Copy link
Contributor

somehowchris commented Apr 26, 2022

Just saw that there is some usage report thing going on for cargo-quickinstall. Mind if I ask, why aren't you tracking too?

  • You could notify maintainers to let them know, people would actually appreciate to use cargo-binstall (kinda like lobbying to get the word spread)
  • You could actually have charts on how much something is downloaded, kinda like the stats crates.io itself presents
  • Maintainers could know which platforms are used, to optimize or add targets to their cd for binary distribution

Over at cargo-all-features I know that there are about 8k recent downloads via crates.io but would there be interest in binaries (aka to be able to use cargo-binstall)? We don't know as we do not have that kind of inisght

@passcod
Copy link
Member

passcod commented Apr 26, 2022

The usage report for quickinstall is because:

  • the quickinstall project directly uses this to build more packages
  • while it wasn't a requirement it was a pretty strong suggestion that it be implemented in order to support quickinstall

I don't personally want to collect tracking information; Ryan may have a different opinion. It's a service to keep up, potentially bills to pay, security to be looked after; there's privacy considerations. It's a very different ballgame to just providing a tool.

Additionally, I don't see the interest in maintaining two tracking services. The features you propose could very well be built into quickinstall's solution; we even have a different user agent so provenance can in theory be differentiated.

Also note that binstall pulls directly from the urls, it doesn't transit or cache, so if you do have binaries up you can get usage stats from those. For github release artifacts, that information is exposed in the github UI (and API).

@ryankurte
Copy link
Collaborator Author

ryankurte commented Apr 30, 2022

yep we're on the same page @passcod, if we can feed better user-agent information (or anything else) to the APIs we're interacting with that could be neat, but definitely prefer no tracking here.

(thinking about feedback, we could have a if this doesn't work for you, please open an issue with the following information link in the help or something to help point people back here when things go wrong?)

@somehowchris
Copy link
Contributor

somehowchris commented May 2, 2022

So, during the last weekend I made some PRs, some intentionally (more or less planed) and some more spontaneousas I ran into some small issues or saw some potential. You @ryankurte engeuraged me to open issues about some things. Just to reiterate, I like this project, so I would like to contribute. Its up to you what you accept, ignore or reject. I'm motivated by learning and also by hearing/reading others opinions.

In order to proceed, I listed some of my wishes/motivations below, I would love to hear from you on which you would like to see PRs, issues or general discussions, or plainly reject. These may include some of the things already merged but may paint the bigger picture

Some ideas:

  • As mentioned above I would love to have a command/flag to install from a Cargo.toml manifest -> This would open the oportunity to version tools for ci/cd or deployments. this would follow along the lines of gitops styles as you could version tooling in git. Specifing multiple crates in the cli would need a redo on how to specify versions.
  • Remove tempdir and use specific extraction for known files feat: use in memory extraction #130 -> no need to make a disk round-trip if the file to extract (or generally an algorithem to compare paths) is known.
  • Move away from openssl as better rust alternatives exist feat: use rustls for crates_io_api #126 feat: remove crates-index dependency #128
  • Self updates or hints to version updates -> npm or pip have a nice message telling if one could update this tooling. This could be nice and easy to implement. Just comparing the version built with the latest version from crates.io. Updating itself would be dangerous but providing a command to do so would benefit dx
  • Aarch64 support Aarch64 support #123 Raspberry pi is going 64bit arm8, so are m1 chips
  • Proper exit codes Proper exit codes #127 -> Currently additional steps need to be added to scripts such as Dockerfiles to check if a binary has been installed. That's not nice and not as easy to use as I believe it should be.
  • Restriction to https transports and package verification Only https for requests #124 Support package signing and verification #1 -> I see https + checksums as the most simple and viable solution. They would make cargo-binstall as secure as cargo-install
  • Support features -> I have seen some comments about this and support for something like this doesn't sound huge. Over at diesel there is an issue to support binary installs, this would be a nice thing. Other rust/cargo cli tools also have feature gates for things like openssl vs rustls,... . A spontaneus idea would be to sort a list of features asc alph + len hash it and introduce a new option for the url templates
  • Either a subcommand or new cli along the lines of cargo-bpackage -> Apply best bractises as I believe many developers of crago crates or rust binaries aren't that much into configuring ci & cd for multiple targets, features. actions-rs made clear that there is space for simple tooling with rust in ci&cd. Developing this as a subcommand or its own binary would give the oportunity to also use this on dev devices as some choose to do releases manually but a gh-action would be nice too. This also would also be a move in the direction of 0 config deploy and use binaries but wouldn't harm existing cd
  • I believe a gh-action to install and run cargo-binstall would be nice and aide some with their multi platform cis to make use of the prebuilt binaries of cargo-binstall
  • Support for third party registries https://doc.rust-lang.org/cargo/reference/registries.html -> This would reintroduce crates-index and would need some investigation
  • Support for authentication against the binary host -> Sometimes not every binary should be public i.e. some binaries for some integration tests
  • Support for targeting git repositories for the Cargo.toml metadata -> I believe this is what Support install directly from git repo #3 references, this would currently reintroduce openssl-sys via git2-rs as gitoxide doesn't seem ready for a bare-clone

More spontaneous ones:

If you feel like discussions or comment threads could be too big, feel free to email me

@passcod
Copy link
Member

passcod commented May 2, 2022

It's great to see enthusiasm for this project! It's a big list and there's certainly a lot of potential. I think primarily what we're looking at moderating here is going too fast and not justifying complexity which will come back to bite us in the long run. One thing in particular you've touched on is this:

If cargo-binstall is being used across hundreds or thousands of ci-pipelines

That would be great success! It would also mean that a breaking change will break thousands of uses. I think we want to avoid that, and a good way to avoid this is to be careful on what we implement now so we are less likely to revisit it later. We can move a tad faster than Rust itself ;) but let's not go too far in the "move fast and break things" direction either.

One thing I see often in open source contributions vs open source maintainership is this idea that Ryan touched on a bit earlier: we want to fix problems that exist. But it's not enough that you have the issue. As maintainers, we're looking at the software as it is used by a lot of people. You having the issue is one person. Sometimes it's obvious that something will affect more people. Sometimes it's much less so. Other times, an issue might be better solved outside of this project, and "fixing" it here is a false herring.

We also want to consider the direction for the project. As a secondary maintainer, this is something that I'll often defer to Ryan about, as it's his project in the first place, right? It's things like, do we want to be a one-stop-shop where you hit cargo binstall package and it figures out the right thing to do, even if it has a bunch of "magic" (yes), and do we want to track users (no). Going forward, it might be do we want to focus on the human experience and discourage CI use or do we want to make both ways as ergonomic as possible or do we want to focus on automated use and near-automated use and perhaps accept some minor annoyances for human use. How much of a "drop in replacement for cargo install" do we want to be? Is that even a goal? That kind of thing.

So, let's take a step back, let's take a breath, let's talk some things out and see where we fall. I think also something to keep in mind is both Ryan and I are fairly busy and don't have a lot of time to dedicate to this project: so it's both great to see progress being made, and a bit overwhelming at the same time.


Over at diesel there is an issue to support binary installs

Can you expand more on this? I can't see how that would fit/work with what diesel is.

@NobodyXu
Copy link
Member

I am working on adding cargo-binstall to taiki-e's install-action taiki-e/install-action#8 so that github action can use cargo-binstall.

@passcod
Copy link
Member

passcod commented Jun 10, 2022

I'm working on the "installing a suite of tools from manifest" thing btw

@yonkeltron
Copy link

I found out about cargo-binstall through this awesome introductory blog post about cargo-dist. I am admittedly late to the party. Unfortunately, when I went to try this out at work, the corporate proxy got in the way and I can't yet figure out how to specify a certificate bundle like I can with other apps. I already have the customary environment variables set properly such as CARGO_HTTP_CAINFO, SSL_CERT_FILE, and SSL_CERT_PATH. This is a bit frustrating but doesn't have to be the end. Excited to learn more.

@kekonn
Copy link

kekonn commented Feb 22, 2023

I found out about cargo-binstall through this awesome introductory blog post about cargo-dist. I am admittedly late to the party. Unfortunately, when I went to try this out at work, the corporate proxy got in the way and I can't yet figure out how to specify a certificate bundle like I can with other apps. I already have the customary environment variables set properly such as CARGO_HTTP_CAINFO, SSL_CERT_FILE, and SSL_CERT_PATH. This is a bit frustrating but doesn't have to be the end. Excited to learn more.

Can confirm this solved it for me as well.

@echoix
Copy link

echoix commented May 20, 2023

It would be nice in the README to have an example of a one-liner to download and install (extract and move) from the links given. At each time I want to use binstall on a new setup without a desktop, I totally forget the syntax when it comes to tar, so I always end up searching how to do it. At least for the linux links

@hawkw
Copy link

hawkw commented Aug 24, 2023

Binaries installed using cargo-binstall will not execute on NixOS, because they have an interpreter path that points to:

interpreter /lib64/ld-linux-x86-64.so.2

but for locally-built executables, the interpreter path is to the ld in the Nix store:

eliza@noctis:~ $ file /home/eliza/.cargo/bin/cargo-binstall
/home/eliza/.cargo/bin/cargo-binstall: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/ibp4camsx1mlllwzh32yyqcq2r2xsy1a-glibc-2.37-8/lib/ld-linux-x86-64.so.2, for GNU/Linux 3.10.0, with debug_info, not stripped

note that the interpreter path is:

interpreter /nix/store/ibp4camsx1mlllwzh32yyqcq2r2xsy1a-glibc-2.37-8/lib/ld-linux-x86-64.so.2

I first encountered this issue while trying to install Oranda using cargo-binstall (axodotdev/oranda#606) , but I'd imagine this is not an Oranda-specific issue and instead occurs whenever using cargo-binstall to install binaries on NixOS.

I'm not sure if there's really anything that cargo-binstall can do to fix this. It's probably better for NixOS users to just install binaries using Nix (I reached for cargo binstall in the hopes that I could install a newer version than what's packaged for Nix). If I'm wrong and there is a way to make it play nice with NixOS, that would be cool, but I imagine the best solution is probably just to document that cargo-binstall doesn't work on NixOS, so that other people don't run into similar problems.

@NobodyXu
Copy link
Member

NobodyXu commented Aug 24, 2023

@hawkw You could use --tarfers to specifically use the musl target instead, that will definitely work.
We can modify crates/detect-targets to disable gnu if the interpreter does not exist or is not executable.

@hawkw
Copy link

hawkw commented Aug 24, 2023

@hawkw You could usw the musl target instead, that will definitely work. We can modify crates/detect-targets to disable gnu if the interpreter does not exist or is not executable.

Oh, that's a great idea, installing statically-linked musl targets on NixOS would probably work fine.

@eitsupi
Copy link

eitsupi commented Aug 27, 2023

You could use --tarfers to specifically use the musl target instead, that will definitely work.
We can modify crates/detect-targets to disable gnu if the interpreter does not exist or is not executable.

Related to this, I recently ran into a problem that I believe was caused by gnu being installed instead of musl. (PRQL/prql#3336)
Could you consider changing the default priority to install musl first?

Thanks for wonderful tool!

@NobodyXu
Copy link
Member

Could you consider changing the default priority to install musl first?

We could add another glibc version check, if it is lower than a certain threshold then we would fallback to musl instead.

We can also provide an environment variable for overriding targets.

@NobodyXu
Copy link
Member

@eitsupi After thinking this again, I think this is more of an issue with mdbook pre-built and it's a good idea to open an issue there.

You could also try using quickinstall build, it uses older glibc.

The best way forward is to check the glibc version used by the binary, if it is newer than the system-installed glibc, then fallback to musl

@eitsupi
Copy link

eitsupi commented Aug 27, 2023

@eitsupi After thinking this again, I think this is more of an issue with mdbook pre-built and it's a good idea to open an issue there.

Thanks for pointing that!
I'm sorry I don't know exactly what the problem is, but perhaps it has something to do with the fact that they are building here using old Ubuntu 20.04?
https://github.com/rust-lang/mdBook/blob/02f3823e4caae28cc0b1ceac284f81743f8fd29b/.github/workflows/deploy.yml#L26-L31

Edit: I found the issue about that. Thanks! rust-lang/mdBook#1954

In any case, a fallback mechanism to musl would definitely be useful.

@Sculas
Copy link
Contributor

Sculas commented Sep 1, 2023

I'm not sure why, but cargo-binstall is really slow for me. I am authenticated to GitHub using gh, so it's not GitHub that's rate-limiting me here. This was me installing cargo-nextest for the first time:

$ cargo binstall cargo-nextest -y
 INFO resolve: Resolving package: 'cargo-nextest'
 WARN The package cargo-nextest v0.9.57 will be downloaded from github.com
 INFO This will install the following binaries:
 INFO   - cargo-nextest.exe (cargo-nextest.exe -> C:\Users\$USER\.cargo\bin\cargo-nextest.exe)
 INFO Installing binaries...
 INFO Done in 196.7039375s

Then, I ran the same command again to check how long it took when cargo-nextest was already installed:

$ cargo binstall cargo-nextest -y
 INFO resolve: Resolving package: 'cargo-nextest'
 INFO resolve: cargo-nextest v0.9.57 is already installed, use --force to override
 INFO Done in 45.3914203s

Taking a look at other issues in this repo, I'm definitely sure that this shouldn't take that long.

@echoix
Copy link

echoix commented Sep 1, 2023

It looks like it is on Windows, can you rule out any antivirus interference in that test?

@Sculas
Copy link
Contributor

Sculas commented Sep 1, 2023

It looks like it is on Windows, can you rule out any antivirus interference in that test?

I first ran cargo uninstall cargo-nextest, then I disabled "Real-time protection" in Windows Defender (I don't have any other AV), and then ran cargo binstall cargo-nextest -y again. These are the results:

$ cargo binstall cargo-nextest -y
 INFO resolve: Resolving package: 'cargo-nextest'
 WARN The package cargo-nextest v0.9.57 will be downloaded from github.com
 INFO This will install the following binaries:
 INFO   - cargo-nextest.exe (cargo-nextest.exe -> C:\Users\$USER\.cargo\bin\cargo-nextest.exe)
 INFO Installing binaries...
 INFO Done in 197.1092732s

No dice, unfortunately. I've reran the same commands, but now with debug logging enabled:

$ cargo binstall cargo-nextest -y --log-level debug

Log: contents.log

@passcod
Copy link
Member

passcod commented Sep 1, 2023

Can you provide that log again but with --json-output and at trace level so we can see where the time spent is? thanks

@Sculas
Copy link
Contributor

Sculas commented Sep 1, 2023

Can you provide that log again but with --json-output and at trace level so we can see where the time spent is? thanks

I've done the same steps as before (uninstall, turn off AV), here's the command I ran:

$ cargo binstall cargo-nextest -y --log-level trace --json-output

Log: contents.log


Looking at the log myself, the 2 things that take the longest are the download from crates.io and the download from GitHub. Why is that? I have a gigabit fiber connection (and a speed test proves that), so it's not my internet being slow. I'm clueless.

@Fyko
Copy link

Fyko commented Oct 19, 2023

Howdy folks! Really enjoying cargo-binstall. I'm not sure if this already exists (because it's not noted in SUPPORT.md), but it would be lovely to support a bin name that's different from the package name. For a project I'm working on, scyllax-cli, I'd like the bin to be named scyllax, which can easily be done with Cargo.toml's [[bin]] feature, but it doesn't seem to work with binstall.

@NobodyXu
Copy link
Member

That's indeed a usecase we yet to support: We assume that the release name contains package name, not binary name.

I'm working on dist-manifest support which might help this, and maybe we can also support this directly in existing fetchers.

@Fyko
Copy link

Fyko commented Oct 20, 2023

I just discovered, it seems to work for taplo-cli.

$ cargo binstall taplo-cli -y
$ which taplo-cli
taplo-cli not found
$ which taplo
/home/carter/.cargo/bin/taplo

@passcod
Copy link
Member

passcod commented Oct 20, 2023

Yes, this should work as we do parse the Cargo.toml. Can you clarify and open an issue with what "doesn't work" in your case?

@ilyagr
Copy link

ilyagr commented Jun 8, 2024

Our project just had a mis-adventure with binstall. It seems that cargo binstall preferred to download binaries from QuickInstall as opposed to our own, and they turned out to be broken.

martinvonz/jj#3844 (comment)
cargo-bins/cargo-quickinstall#250

Is there a way to specify in our Cargo.toml that our binaries should be preferred? We already have some cargo binstall config:

https://github.com/martinvonz/jj/blob/65a988e3d2d6b3ed201ed8c2b4952ef08344e55f/cli/Cargo.toml#L118-L121

Another issue is that we only provide musl binaries, so perhaps binstall prefers QuickInstall because it has gnu binaries.

@NobodyXu
Copy link
Member

NobodyXu commented Jun 9, 2024

Another issue is that we only provide musl binaries, so perhaps binstall prefers QuickInstall because it has gnu binaries

Yes that's why quickinstall is preferred, on gnu target it will try crate-meta-data => quickinstall, if it fails, then fallback to musl.

We can add a new configuration key to Cargo.toml to disable quickinstall fallback, for now you can either

  • use --disable-strategies quick-install to disable quickinstall
  • provide a gnu binary

Thanks for reporting this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests