Python instead of shell

[tcharding]

I like nerding out and writing shell scripts as much as the next bloke but it is a time sink and there are always bugs. #2633 just uncovered a bunch of things we were doing wrong, and even with a chat bot it took me ages to fix them.

Is there a reason we don't use Python for all our test infrastructure?

[tcharding]

I thought @RCasatta attempted to introduce Python somewhere before hand but I don't see it.

[RCasatta]

I don't think it was me who wanted to introduce Python... But yeah I don't like bash scripts I think it should be all rust

[apoelstra]

Python is super annoying to use to do shell-like things like search-and-replace, reading from files and splicing their contents, shelling out to git, etc.

I am also not very familiar with Python (though the situation there is certainly better than with Bash).

[dpc]

I find it Python still quite error prone and not very productive. The upgrade in expressiveness fails to make up for downgrade in "shellness", IMO. Which combined with introducing another language to the project, makes me never consider it.

Historically "just running Python" was never as easy, and dependency incompatibility, bugs etc. made me dread using software implemented in it. It might have gotten better with Poetry and proper lock file handling, but I'm forever averse to using anything in Python.

https://docs.rs/duct/latest/duct/
https://docs.rs/xshell/latest/xshell/

Writing scripts in Rust is quite productive, and does not introduce another language. The biggest challenge is making sure they are available compiled for the end user (developer). It's not very unlike a different problem with Python - you need to make sure the user has a Python in the PATH (which I don't).

If you have something like a Nix dev shell, you can just seamlessly compile all Rust scripts and add them to the PATH. If you don't, you might have a just command that does it (part of some kind of just setup that compiles them and dumps them into ./bin/ or something). Alternatively the scripts are a sub-crate of the workspace that compiles binaries to ./target/dev and then other scripts just assume it's there.

Another approach is cargo-script rust-lang/rfcs#3502 which hopefully will be stable ... in not too distant future. Then it would just come with cargo for free, which would be great. Unfortunately for me this approach is not very composable with Nix sandbox, and there's currently no good Nix solution to this problem in Nix land.

So currently in Fedimnint we write shell scripts with set -euo pipefail for anything doesn't feel too complex (majority of small utilities). Once it does feel too complex it is ripped out, RIIRed into another git repo, and compiled as a binary by the Nix shell.

[junderw]

Wouldn't it be as simple as adding "contrib" to the root Cargo.toml workspaces, and place a new Cargo.toml inside contrib, then replace each .sh with an .rs file and make them a [[bin]] in the contrib/Cago.toml?

Ok, maybe not simple... but I don't know why we can't assume the user has cargo installed?

cargo run -p contrib --bin run_task

[dpc]

One might wrap every Rust-script invocation with an outer shell script that either does the compilation and adds target dir to PATH or calls cargo run, sure. It works and is probably good enough. It's reasonable to assume cargo.

[elichai]

Personally I really like the https://github.com/matklad/cargo-xtask pattern/trick for a customized build system

[apoelstra]

Ok, maybe not simple... but I don't know why we can't assume the user has cargo installed?

Cargo is fine. The hard part is network access to find and install a bunch of crates that are unrelated to the crate being tested.

[elichai]

Ok, maybe not simple... but I don't know why we can't assume the user has cargo installed?

Cargo is fine. The hard part is network access to find and install a bunch of crates that are unrelated to the crate being tested.

Note that xtask itself doesn't require new reps, just adding a binary to the workspace,
Using xshell and friends for easier scripting does require unrelated deps though

[junderw]

I think we should write them in Rust instead.

Perhaps we can do something like CARGO_TARGET_DIR=./contrib/.script_target_dir to keep the dependencies for the scripts separate and easier to clean up.