async traits

[junderw]

Starting a discussion on async traits.

Two topics of discussion I'd like to start with:

1. Is the async_trait crate acceptable or should we wait until we bump MSRV to >= 1.75?
2. What are some APIs that would benefit from offering an async version of the trait?

I think for 1 that waiting a few years for MSRV to get high enough is probably desirable.

For 2, I think signing related APIs tend to have use cases for async/await since hardware wallets etc. wait on user input and wallet wait on the hardware wallets.

[Kixunil]

I think async traits are entirely unnecessary in the bitcoin crate. We're not doing IO except for serialization/deserialization, which will be abstracted over sync/async ⨯ std/no_std using push_decode - see #2184

The only one which may need this is bitcoincore-rpc but I'm not sure if async traits are needed there either.

[junderw]

We're not doing IO

But the wallet using the bitcoin crate might.

What if someone has a wallet that needs to send the data over a socket to some HSM?

They need to wait for the HSM response in that case. This is IO.

In BitcoinJS we handle this use case by accepting a Signer argument for signInput and a SignerAsync argument for signInputAsync etc. This allows for those people to implement some class that implements the SignerAsync interface and pass it in. We don't care about the implementation.

Something like that would be useful in Rust as well. An abstraction over signing, which PrivateKey and Xpriv would implement. Then, alternatively, an async version that people could choose to implement for their IO based signing setup...

Not sure if these kinds of abstractions should be higher up in something like bdk, but since a lot of HSMs are pretty low level and only concern themselves with signing hashes/preimages, it seems like the abstraction should be pretty low.

[Kixunil]

But the wallet using the bitcoin crate might.

Doesn't matter, async code can call non-blocking sync code and all our sync code is non-blocking.

What if someone has a wallet that needs to send the data over a socket to some HSM?

They just split the calls like all other Rust code does:

let foo = bitcoin::something();
let bar = hsm.call_with_foo(foo).await;
let baz = bitcoin::use_bar(bar);

There's no problem with that.

An abstraction over signing, which PrivateKey and Xpriv would implement.

That really shouldn't be required, the crate already provides sufficiently low-level primitives to not need that and the Sign trait proposed in #2405 is just for a convenience method. Just write two more lines, no big deal. And if you want to avoid those two lines, just define your own trait and one day when MSRV is fine, it can be added. Note that I really don't want to abstract over async unless there's full language support, so it'd be just a new trait and new methods.

Anyway I've made an issue for you to make sure nothing slips through: #2409

[dpc]

Async is an unfinished Rust flavor that is a PITA to use and I'd advise everyone to avoid it as much as they can. If you're interested about what I think about it: you can take a look. There is no way in Rust to be generic over async, so you'll inevitability having to deal with worst of both worlds if you want to support both, which means you'll just support async like most of the ecosystem does already: https://www.reddit.com/r/rust/comments/197811x/the_bane_of_my_existence_supporting_both_async/

IMO rust-bitcoin should avoid it as much as possible by being IO-agnostic. If software needs to read and deserialize in an async context, it should just read into memory first, then deserialize later, or figure out something else (push based encoding). Maybe there's a room for separate crate that does consensus encoding on async io traits (are these even stable thing yet?).

[dpc]

Do I read it right, that's it's basically streamlined mio?

[Kixunil]

@dr-orlovsky I meant a piece of simple code that uses it as a demo. Like an echo server.

[dr-orlovsky]

I think https://github.com/cyphernet-dao/nsh can be seen as a simple sample project using io-reactor. Specifically, you can look here: https://github.com/cyphernet-dao/nsh/blob/master/src/server.rs

It is an analog of SSH tool, but (a) using Noise framework instead of OpenSSH for the encryption and (b) using io-reactor

PS. Here is the echo command you was asking about: https://github.com/cyphernet-dao/nsh/blob/836323230693bbad4fc3d508c12d35f7e8c1b7e6/src/processor.rs#L76-L93

[Kixunil]

Cool, I think I understand it now. Two things jumped at me:

• There's quite a bit of boilerplate because all methods of Handler are mandatory. I think the common case is just IO handling, not timers.
• You pass RawFd in public API which clearly excludes Windows support (I wouldn't need it but some people might).

[dr-orlovsky]

Regarding (1) I think you are right and there can be a default implementation doing nothing.

Regarding the (2) you are not oblidged to use RawFd; it is just the approach taken by the demo

But yes, reactor is not yet windows-compatible, that is one of the tasks I plan to address in some foreseable future.

[dpc]

Well, that's the problem, you really can't. If your browser is stuck waiting for data to arrive the syscall just waits "forever" (5 min).

Whatever async IO is using under the hood, can be done in blocking IO code to enforce a deadline on a timer / select on a flag. It would have been slightly more "manual", but Rust is very good at building abstractions. And even if slightly cumbersome, it would only needed to be used where actually needed, instead of turning the language inside out, breaking composability of the ecosystem and infecting even code that doesn't care about it with all the async complexity.

And ... if someone really needs it, they should only do that in async code, in a limited scope. I don't dislike async as much as I dislike how prevalent it became. For decades software of all kinds was built without async and suddenly it's "async, async, async" everywhere.

[Kixunil]

Well, maybe if the ecosystem is async anyway it's better to use it then reinvent cancellation. I don't hate async either, I think most of its problems are just language shortcomings (like lack of GAT).

[dpc]

Exactly. There's bunch of problems, though lots of them theoretically fixable. But it might take years, while the community happily jumped on the bandwagon 2 years ago.

[vincenzopalazzo]

idk if we can achieve something without implementing a PoC of this, it is possible to implement the async trait and expose them with a feature flag anyway.

The async in rust will be out iirc on 2027 for real usage, there are too many feature that are missing like async drop and dyn*

[Kixunil]

As was discussed this crate doesn't really need async traits, we can workaround it nicely. However, it'd really help if we had Future<Context> which I've seen proposed somewhere (there was an article on reddit linking to it recently), since then I could write a nice hack for push_decode. (Although I would have to figure out what to do about pinning which is another can of worms.)