Crate Smashing

[tcharding]

Let the smashing begin.

Crate smashing has been discussed in a few places but the discussion is old. We have flattened util now, lets start the discussion of crate smashing anew with the current "clean" state.

Some previous discussion threads:

• Rearrange all the modules #525
• Consider flattening util module? #639
• Proposal: split bitcoin into multi-crate workspace #550

The task is quite complicated to model so to assist us I have created a demo repo that can be used to test things out, currently it has a skeleton of the current state and a possible crate bitcoin-str (incl. subcrates). I've only added types to check that dependencies work for the bitcoin-str crate.

• https://github.com/tcharding/rust-bitcoin-crate-smashing-demo

The point of the demo repo was to avoid what happened in #550 with a million different pictures of the crate tree and no clear idea on what to work towards. Please holla if you have a better idea of how to brain storm this. I envision adding branches to the demo repo to test out ideas in parallel.

[tcharding]

I've invited all maintainers or rust-bitcoin to join the demo crate so you have write access and can push branches without having to do PRs - if you so wish.

[tcharding]

Possibly worth discussing splitting rust-miniscript up here as well: #653

[Kixunil]

Honestly, I find images more clear. :)

I think that at #550 we concluded we don't need bitcoin-str? It was originally meant for hex and other encodings but

• There's a circular dependency with hex-bitcoin_hashes-base58
• Depending on e.g. base58 looks silly if you don't use Address (e.g. electrs doesn't)

We determined that putting hex into internals and exposing it via Display would work.

[apoelstra]

For users that want to hex-encode their own types, having to route their code through the Display impl of bitcoin types (even if we exposed a Hexed(&[u8]) say) would be pretty annoying.

[Kixunil]

If we're going to provide general-purpose hex features then I believe we should have a separate, properly-maintained, crate for that.

[apoelstra]

Like, separate from all the other common encodings that we use in Bitcoin? Why? This would mean that we're blocked on getting a proper crev process going in order to have basic functionality we've been waiting 10+ years for stdlib to provide.

Right now we have bitcoin_hashes::hex. This is properly maintained. It works fine. But it comes with a bunch of hashes and doesn't come with base64, which doesn't make a lot of logical sense. Hence bitcoin-str.

[Kixunil]

The other encodings are only common if you make certain kinds of applications. But I guess there's not much harm if bitcoin_str only has hex and base64. And we should get crev going regardless of whether the crate is separate or not.

[apoelstra]

That's a fair point -- hex and base64 are pretty common "systems programming" encoding (hex is used everywhere, base64 in pgp and http among other things) -- but base58, bech32 etc., are all very bitcoin specific. So there's a reasonable argument that hex and base64 should get their own crates which try to be generally applicable and well-maintained with that goal in mind.

But because they're both so simple, especially hex, it really pains me to add it as a an independent dependency, and I'm having trouble getting past that (even with crev).

Definitely agreed that we should prioritize crev regardless of what we do here.

[tcharding]

There's a circular dependency with hex-bitcoin_hashes-base58

The reason I originally did that demo crate was to prove there is no dependency cycle, unless I'm missing something it proves that.

[tcharding]

I had in mind the following (all names are crates except base64) with bitcoin- prepended as needed so they can be released on crates.io). foo -> bar means foo depends on bar.

bitcoin -> str

str -> hex
    -> base58
    -> bech32
    -> base64 (could also be a module instead of a crate)

base58 -> hashes
       -> hex

hashes -> hex

All crates re-exported by str can be optional, bitcoin would then have to enable all of them. If for some reason (that I don't understand) we do not want to release a hex crate then that stuff could go in internals.

Or am I missing something that you guys are saying?

[Kixunil]

I worry that having another facade crate (str) will add maintenance overhead without much value. What's wrong with just depending on the individual crates directly?

[apoelstra]

I agree with @Kixunil, and am even starting to warm up to his "everything gets a crate rather than being modules in str" model.

It also gives us a nice pathway forward to un-NIH-ing base64 and hex, if we find common ground on MSRV and crev their code (which I think are both likely, in the future).

The argument against this is "why the hell do we need an extra dep just for hex, it's like 2 LOC", but (a) it's not two LOC if you do it efficiently and (b) the reason why is that the stdlib shafted us, we just have to deal with it and (c) the "solution" we've chosen, of just shoving hex into a random existing dependency, reduces the size of cargo tree but morally it actually makes the dependency trust situation worse.

[Kixunil]

it's not two LOC if you do it efficiently

Exactly, it's more involved than one would expect and could even potentially include unsafe (even though we decided not to yet). And the perf difference is really huge. Also I think it should be possible to reduce monomorphisation cost by sharing efficient impl for 32B arrays.

but morally it actually makes the dependency trust situation worse.

Because it makes it harder to review since the crate contains a bunch of unrelated hashing code? Or do you have something else in mind?

[apoelstra]

Because it makes it harder to review since the crate contains a bunch of unrelated hashing code? Or do you have something else in mind?

Yep, exactly. Plus, putting hex into the hashing crate signals "this is meant specifically for hashing and might break if you use it elsewhere" and/or "these guys don't know wtf they're doing :)".

[tcharding]

Crate smashing might be a complex enough that a call to whiteboard it would be useful?

[apoelstra]

Yes, though I think we still have a few disagreements to hash out about the granularity of crate smashing, which we could do before whiteboarding out a complete, detailed solution.

[Kixunil]

I actually think that the attempt at doing it was more productive than endless discussions we had before. Lot of things cleared up and I suspect even more will in the future.

[tcharding]

Perhaps I'll surge forward with PRs then. They can be part "this is what we discussed" and part exploration with the hope of finding new/better ways.

[Kixunil]

Sure, I was already looking at separating consensus into a crate (so that types can independently impl it) a few months ago. I think this will be required soon. But there will be an intermediate step of moving some hex-related stuff to internals and depend on it from consensus.

[tcharding]

This PR has relevant discussion in it: #2466