Multiprocess bitcoin

[ryanofsky]

This is a draft PR because it is based on #28921 and #28929. The non-base commits are:

• ee4b9138c837 doc: multiprocess documentation improvements
• df27e301e603 Increase feature_block.py and feature_taproot.py timeouts
• cbc5e366a216 Add util::Result workaround to be compatible with libmultiprocess
• 3427e7e0a965 Add capnp serialization code for bitcoin types
• b6516c454f70 Add capnp wrapper for Handler interface
• 5cf30e5357fc Add capnp wrapper for Chain interface
• b5fd425e075a Add capnp wrapper for Wallet interface
• 329b34c85ac6 Add capnp wrapper for Node interface
• dde2a2211038 Make bitcoin-gui spawn a bitcoin-node process
• f3349a24e5ac Make bitcoin-node spawn a bitcoin-wallet process
• 6ea638ca4f28 multiprocess: Add debug.log .wallet/.gui suffixes
• 622acd03b043 doc: Multiprocess misc doc and comment updates
• 755ca62696a4 combine_logs: Handle multiprocess wallet log files
• 56ef459b5734 Update libmultiprocess library

---

This PR adds an --enable-multiprocess configure option which builds new bitcoin-node, bitcoin-wallet, and bitcoin-gui executables with relevant functionality isolated into different processes. See doc/design/multiprocess.md for usage details and future plans.

The change is implemented by adding a new Init interface that spawns new wallet and node subprocesses that can be controlled over a socketpair by calling Node, Wallet, and ChainClient methods. When running with split processes, you can see the IPC messages going back and forth in -debug=1 output. Followup PR's #19460 and #19461 add -ipcbind and -ipcconnect options that allow more flexibility in how processes are connected.

The IPC protocol used is Cap'n Proto, but this could be swapped out for another protocol. Cap'n Proto types and libraries are only accessed in the src/ipc/capnp/ directory, and not in any public headers or other parts of bitcoin code.

Slides from a presentation describing this change are available on google drive. Demo code used in the presentation was from an older version this PR (tag ipc.21, commits).

---

This PR is part of the process separation project.

[jonasschnelli]

Oh. Nice.
I expected much more code to achieve this.

Conceptually I think this goes into the right direction, though, I'm not sure if this could end up being only a temporary in-between step that may end up being replaced.
Because, it may be more effective to split the Qt/d part completely and let them communicate over the p2p protocol (SPV and eventually RPC). More effective because it would also allow to run Qt independent from a trusted full node (if not trusted, use mechanism like full block SPV, etc.).

Though, I'm aware that capnp has an RPC layer. But this would introduce another API (RPC / ZMQ / REST and then capnp RPC).

I'm not saying this is the wrong direction, but we should be careful about adding another API.

Three questions:

• Would the performance be impractical if we would try to use the existing RPC API?
• Could the capnp approach (or lets say IPC approach) be designed as a (or the) new API ("JSON RPC v2" and replacement for ZMQ)?
• Does capnp provide a basic form of authentication? Would that even be required?

[ryanofsky]

Would the performance be impractical if we would try to use the existing RPC API?

Reason this is currently using capnp is not performance but convenience. Capnp provides a high level API that supports bidirectional, synchronous, and asynchronous calls out of the box and allows me to easily explore implementation choices in bitcoin-qt without having to worry about low level protocol details, write a lot of parameter packing/unpacking boilerplate, and implement things like long polling.

Capnp could definitely be replaced by JSON-RPC, though, and I've gone out of my way to support this by not calling capnp functions or using capnp types or headers anywhere except the ipc/server.cpp and ipc/client.cpp files. No code outside of these two files has to change in order to move to a different protocol.

Could the capnp approach (or lets say IPC approach) be designed as a (or the) new API ("JSON RPC v2" and replacement for ZMQ)?

It could, but I'm going out of my way right now specifically NOT to add yet another bitcoind public API that could add to the JSON-RPC/REST/ZMQ/-blocknotify/-walletnotify confusion. The IPC here doesn't happen over a TCP port or even a unix socket path but over an anonymous socketpair using an inherited file descriptor. (I haven't done a windows implementation yet but similar things are possible there).

I'm trying to make the change completely internal for now and transparent to users. Bitcoin-qt should still be invoked the same way and behave the same way as before, starting its own node and wallet. It just will happen to do this internally now by forking a bitcoind executable rather than calling in-process functions.

This change will not add any new command line or GUI options allowing bitcoin-qt to connect to bitcoinds other than the one it spawns internally. Adding these features and supporting new public APIs might be things we want to do in the future, but they would involve downsides and complications that I'm trying to avoid here.

Does capnp provide a basic form of authentication? Would that even be required?

It's not required here because this change doesn't expose any new socket or endpoint, but it could be supported. Capnp's security model is based on capabilities, so to add authentication, you would just define a factory function that takes credentials as parameters and returns a reference to an object exposing the appropriate functionality.

[gmaxwell]

I'm really uncomfortable with using capn proto, but fine enough for some example testing stuff!

I'm a fan of this general approach (ignoring the use of capn proto) and I think we should have done something like it a long time ago.

[dcousens]

strong concept ACK, but if is feasible, would prefer usage of the existing RPC instead of capn'proto

[laanwj]

Concept ACK, nice.

I'm really uncomfortable with using capn proto, but fine enough for some example testing stuff!

Please, let's not turn this into a discussion of serialization and RPC frameworks. To be honest that's been one of the things that's putting me off of doing work like this. If you want to suggest what framework to use, please make a thorough investigation of what method would be best to use for our specific use case, and propose that, but let's not start throwing random "I'm not comfortable with X" comments.

We already use google protocol buffers in the GUI for payment requests to in a way that would be the straightforward choice. I'm also happy you didn't choose some XML-based abomonation or ASN.1. But anyhow, not here. For this pull it's fine to use whatever RPC mechanism you're comfortable with.

This change will not add any new command line or GUI options allowing bitcoin-qt to connect to bitcoinds other than the one it spawns internally.

I'm also perfectly fine with keeping the scope here to "communication between GUI and bitcoind". This is not the place for introducing another external interface. Might be an option at some point in the future, but for now process isolation is enough motivation.

[ryanofsky]

Updated and rebased bf5f8ed -> 0ca73bc (pr/ipc.0 -> pr/ipc.1)

There's a lot of new changes here. More functions and callbacks have been wrapped, and there's now support for asynchronous calls that don't block event threads in the client and server.

At this point it would be very helpful to have code review for the main commit (0ca73bc "Add barebones IPC framework to bitcoin-qt and bitcoind"), because all the real infrastructure is now in place, and the main thing left to do is wrap up more functions and callbacks in IPC calls so the GUI can be functional.

[ryanofsky]

Updated and rebased 0ca73bc -> 5e28c2f (pr/ipc.1 -> pr/ipc.3) to avoid a conflict. Main addition is an expanded src/ipc/README.md file.

Again it would be very helpful to have some code review for the main commit (5e28c2f "Add barebones IPC framework to bitcoin-qt and bitcoind"). Giving feedback on the README file would be an easy place to start.

[ryanofsky]

Updated 5e28c2f -> dda3756 (pr/ipc.3 -> pr/ipc.4)

This implements two suggestions from @JeremyRubin:

• It includes a small commit demonstrating what it looks like to add a single new method to the API:
  dda3756 Add ipc::Node::getNodeCount method. This should help give a clearer picture of the layers involved in implementing an IPC call.

• Instead of adding Cap'n Proto code and modifying Qt code in a single commit, it includes a new early commit (1407a2b Add ipc::Node and ipc::Wallet interfaces that introduces new src/ipc/interfaces.h and src/ipc/interfaces.cpp files and ports Qt code to use them without any Cap'n Proto stuff. This shows the separation between Qt updates and IPC implementation details better and makes it easier to see how a different IPC system could be substituted in for Cap'n Proto. This commit could even be made into a separate PR.

[ryanofsky]

@laanwj pointed out in IRC (https://botbot.me/freenode/bitcoin-core-dev/msg/83983170/) that this change could help make the GUI more responsive by preventing Qt event processing from getting blocked, which currently happens in the monolithic bitcoin-qt when the main GUI thread makes a call to a slow libbitcoin function, or waits a long time for a cs_main or cs_wallet lock.

At the time in IRC, I didn't think this change could directly help gui responsiveness, because although it does move libbitcoin and LOCK calls out of the bitcoin-qt process and into the bitcoind process, it just replaces these calls with blocking IPCs that make the GUI equally unresponsive when they tie up the main GUI thread.

However, this doesn't have to be the case. The place where IPC calls currently block waiting for responses is the return promise.get_future().get(); line in ipc::util::Call::send method here: https://github.com/ryanofsky/bitcoin/blob/pr/ipc.4/src/ipc/util.h#L166

But the std::promise object used in that line could easily be replaced with a Qt-aware promise object that processes GUI events while the promise is blocked. (The Qt-aware promise implementation would check if it is being used on the main GUI thread, and if so use a local Qt event loop substituting
loop.exec() for std::future::get() and loop.quit() for std::promise::set_value().)

This would add more overhead and make the average IPC call a little slower. But it would avoid situations where an unexpectedly slow IPC call ties up the whole gui, so it might be worth doing anyway.

[laanwj]

@ryanofsky Yes, integrating the IPC event loop and Qt event loop would help responsiveness.
Though I remember there were some issues in some cases with recursively calling into the Qt event loop (e.g. things need to be reentrant, deleteLater stuff runs earlier than expected, to keep in mind).

[sipa]

@ryanofsky I'm not familiar with Qt or capnproto, but I don't understand what the move to a different process has to do with making things less blocking. Any changes in architecture that would result in less blocks should equally be possible within the same process.

[sipa]

This change will not add any new command line or GUI options allowing bitcoin-qt to connect to bitcoinds other than the one it spawns internally. Adding these features and supporting new public APIs might be things we want to do in the future, but they would involve downsides and complications that I'm trying to avoid here.

I don't understand the goal here. On itself, there seems little benefit in separating the GUI and the rest into separate processes if those two processes still depend on each other (this is different from separating the wallet from the node, for example, as there as security considerations there... but for that use case the easiest approach seems to just have a lightweight mode and running two instances).

I think it would be awesome if bitcoin-qt could be started and stopped independently to control a bitcoind process in the background, but if that's not the intent, what is the purpose?

[ryanofsky]

Any changes in architecture that would result in less blocks should equally be possible within the same process.

Let's say there are 50 places where bitcoin-qt calls a libbitcoin function. That means there are 50 places to update if you want bitcoin-qt handle to events while the function calls are executing. WIth the IPC framework, there is only one place you have to update instead of 50 places (if you want to do this).

On itself, there seems little benefit in separating the GUI and the rest into separate processes if those two processes still depend on each other.

Ok, so you think the benefits are small, and I think they are more significant.

I think it would be awesome if bitcoin-qt could be started and stopped independently to control a bitcoind process in the background,

This is trivial once bitcoin-qt is controlling bitcoind across a socket. I'm just implementing the socket part first, without introducing new UI features for now.

[sipa]

I think it would be awesome if bitcoin-qt could be started and stopped independently to control a bitcoind process in the background,

This is trivial once bitcoin-qt is controlling bitcoind across a socket. I'm just implementing the socket part first, without introducing new UI features for now.

Ok, that's what I was missing. It wasn't clear to me that this was a just first step towards a more useful separation.

[DrahtBot]

🐙 This pull request conflicts with the target branch and needs rebase.

[DrahtBot]

⌛ There hasn't been much activity lately and the patch still needs rebase. What is the status here?

• Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
• Is it no longer relevant? ➡️ Please close.
• Did the author lose interest or time to work on this? ➡️ Please close it and mark it 'Up for grabs' with the label, so that it can be picked up in the future.

[willcl-ark]

I've been using this in a side-project and noticed some unintended behaviour, which could be my system; if I make distclean (removing the generated files) and then make -j16 the build fails as it can't find the built artefacts. Running plain make works fine every time, and sometimes a low job number works too.

The Makefile appears (to me) to have the dependencies listed correctly, so I'm not entirely sure what's going wrong, but just thought I'd report here in case anyone else had a similar issue. Also, likely not worth actually fixing now before a move to CMake.

[willcl-ark]

I noticed some other unexpected behaviour when interacting with bitcoin-node from an external process which might be of interest to investigate...

I made a (malformed) call to MakeWalletLoader, not including GlobalArgs, which cause the node to shut down whilst it had an open lock on another (default loaded) wallet. This resulted in preventing this wallet from being re-opened again at next startup.

I wonder whether a malformed call to MakeWalletLoader should shut down the node at all, but if it should, then it seems like (perhaps) we should take care to close opened wallets more carefully?

Log

2024-05-22T13:49:06Z [ipc] {bitcoin-wallet-3127478/bitcoin-wallet-3127478} IPC server post request  #3075 {bitcoin-wallet-3127478/bitcoin-wallet-3127902 (from bitcoin-node-3127476/b-scheduler-3127477)}
2024-05-22T13:49:06Z [ipc] {bitcoin-wallet-3127478/bitcoin-wallet-3127478} IPC server send response #3075 ChainNotifications.transactionAddedToMempool$Results ()
2024-05-22T13:49:06Z [ipc] {bitcoin-node-3127476/b-scheduler-3127477} IPC client recv ChainNotifications.transactionAddedToMempool$Results ()
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server recv request  #201 Init.construct$Params ()
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server send response #201 Init.construct$Results (threadMap = <external capability>)
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server recv request  #202 Init.makeEcho$Params (context = (thread = <external capability>))
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server post request  #202 {bitcoin-node-3127476/b-capnp-loop-3210934 (from )}
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server send response #202 Init.makeEcho$Results (result = <external capability>)
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server recv request  #203 Echo.echo$Params (context = (thread = <external capability>), echo = "Hello, world!")
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server post request  #203 {bitcoin-node-3127476/b-capnp-loop-3210934 (from )}
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server send response #203 Echo.echo$Results (result = "Hello, world!")
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server recv request  #204 Init.makeChain$Params (context = (thread = <external capability>))
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server post request  #204 {bitcoin-node-3127476/b-capnp-loop-3210934 (from )}
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server send response #204 Init.makeChain$Results (result = <external capability>)
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server recv request  #205 Chain.getHeight$Params (context = (thread = <external capability>))
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server post request  #205 {bitcoin-node-3127476/b-capnp-loop-3210934 (from )}
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server send response #205 Chain.getHeight$Results (result = 196713, hasResult = true)
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server recv request  #206 Chain.getSettingsList$Params (context = (thread = <external capability>))
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server post request  #206 {bitcoin-node-3127476/b-capnp-loop-3210934 (from )}
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server send response #206 Chain.getSettingsList$Results (result = [])
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server recv request  #207 Init.makeWalletLoader$Params (context = (thread = <external capability>))
2024-05-22T13:49:15Z [ipc] {bitcoin-node-3127476/b-capnp-loop-3127479} IPC server post request  #207 {bitcoin-node-3127476/b-capnp-loop-3210934 (from )}
bitcoin-node: common/args.cpp:728: void ArgsManager::SetConfigFilePath(fs::path): Assertion `!m_config_path' failed.

Note, I haven't tried to reproduce this, but the above is what I think happened.