NotEnoughSpendableUTXOs error for mainnet federation with sufficient UTXOs

[bradleystachurski]

A mainnet federation is getting a NotEnoughSpendableUTXO error when attempting to withdraw on-chain. I've spoken out-of-band with a few of the guardians and we've accumulated some clues:

• Federation on v0.2.2
• 3 successful peg-ins
• 2 successful withdrawals
• Subsequent withdrawal requests show NotEnoughSpendableUTXO
• Using fedimint-observer, we've identified the federation has 3 UTXOs with sufficient balance for subsequent withdrawal requests
• fedimintd logs show repeated Broadcasting pending transactions
• bitcoind backend running in pruned mode (550Mb)

Given this set of information, the current working theory is the wallet is unable to recognize change outputs from the previous two successful withdrawals. In the wallet server when we sync blocks up to consensus height, we call get_tx_block_height for all pending transactions each block.

fedimint/modules/fedimint-wallet-server/src/lib.rs

Line 999 in fa218de

.get_tx_block_height(txid)

If a peer is using bitcoind for a backend, this calls getrawtransaction.

By default this function only works for mempool transactions. ... When called without a blockhash argument, getrawtransaction will return the transaction if it is in the mempool, or if -txindex is enabled and the transaction is in a block in the blockchain.

This implies that if a peer is running bitcoind in pruned mode without enabling txindex, it will only query transactions stored in the mempool. Since the wallet syncs blocks up to consensus height once there's enough confirmations (10 blocks), the call to get_tx_block_height returns nothing since the transaction has been removed from the mempool for some time.

To test this theory, the guardians of this federation are going to attempt switching backends to esplora. If this is successful, this implies a material limitation to using a pruned bitcoind node for peers.

I'll update this issue as I learn more.

[elsirion]

This looks like a potentially very severe bug. The get_tx_block_height approach sounds dangerous, I'd just get all transaction hashes of the block we are processing and then scan for inclusion of our change generating transactions. That would kill the electrum backend, but is anyone actually using it?

[joschisan]

It seems to me that when the rpc call to get_tx_block_height happens to fail the change output is just not recognized. That would cause a consensus failure.

[bradleystachurski]

It seems to me that when the rpc call to get_tx_block_height happens to fail the change output is just not recognized. That would cause a consensus failure.

@joschisan I think your interpretation is correct. @elsirion and I discussed out-of-band a high level approach that would force a rescan of already processed block heights that would recognize change from pending transactions. Do you see any problems with that approach?

[joschisan]

@bradleystachurski Not on a high level, ping me for a review.