SPV multi-server cross-validation

[shesek]

Also includes a bunch of other small changes/fixups.

This PR depends on:

• A fix to Blockstream/electrs that isn't yet deployed to the live servers. The GitLab CI was updated to use the updated version of electrs. (8ddc1c8)
  This was deployed.

• A rust-bitcoin fork with serializable uint types. This is needed to store the validation result in the cache. A PR was sent upstream: Make uint types (un)serializable rust-bitcoin/rust-bitcoin#511 Merged.

• A rust-electrum-client fork with support for using custom TcpStreams. This is needed to set timeouts. A PR was sent upstream: Allow to initialize SSL clients with an existing TcpStream bitcoindevkit/rust-electrum-client#25
  Upgraded to upstream rust-electrum-client v0.4.0 with the necessary changes.

[shesek]

I added a new SPVVerifyResult::NotLongest status. Its encoded as the string not_longest or as the number 4. When spv cross-validation fails, all of the transactions will report this status.

Should this status also be exposed as a global flag of some sort, or is having this on the individual transactions sufficient?

The cross-validator reports some information about the chain fork, including how far behind the primary electrum server is compared to the longest chain (both work and height wise). Should this information be made available to the app?

[shesek]

I added a script for updating the Electrum servers list from the latest Electrum release: https://github.com/shesek/gdk/blob/202010-spv-crossvalidation/subprojects/gdk_rust/gdk_electrum/scripts/update-server-list.sh

Should this be added to some build process somewhere?

[shesek]

Rebased on master and removed the dependency on my rust-electrum-client fork (the necessary changes were added upstream and released in v0.4.0). It still depends on my rust-bitcoin fork pending a PR I sent there.

Other standing items/questions:

• The apps need to handle the new NotLongest and Unconfirmed SPV verification statuses, as well as support configuring the new spv_cross_validation and spv_cross_validation_servers options.

• Should the SPV cross-validation status be exposed as global session state, in addition to being included on individual transactions?

  This could be useful for displaying cross-validation failures somewhere prominent (outside of individual txs), and could also provide the app some additional details about the chain fork, like the local height, the height of the best known chain and the additional work in the best chain.

• Cross-validation is still not implemented in the C FFI used by the Green apps for SPV verification. In that environment, the Rust side doesn't run background threads of its own, it only runs when invoked from the CPP side. This introduces some complexities, both because cross-validation is an ongoing process and because the secondary servers are likely to be slow (or entirely unresponsive), which could result in user-visible latency if its not done in a separate thread.

[LeoComandini]

Rebased on master and removed the dependency on my rust-electrum-client fork (the necessary changes were added upstream and released in v0.4.0). It still depends on my rust-bitcoin fork pending a PR I sent there.

Other standing items/questions:

* The apps need to handle the new `NotLongest` and `Unconfirmed` SPV verification statuses, as well as support configuring the new `spv_cross_validation` and `spv_cross_validation_servers` options.

* Should the SPV cross-validation status be exposed as global session state, in addition to being included on individual transactions?
  This could be useful for displaying cross-validation failures somewhere prominent (outside of individual txs), and could also provide the app some additional details about the chain fork, like the local height, the height of the best known chain and the additional work in the best chain.

* Cross-validation is still not implemented in the C FFI used by the Green apps for SPV verification. In that environment, the Rust side doesn't run background threads of its own, it only runs when invoked from the CPP side. This introduces some complexities, both because cross-validation is an ongoing process and because the secondary servers are likely to be slow (or entirely unresponsive), which could result in user-visible latency if its not done in a separate thread.

Currently no production app is using SPV verification. The points you raise are reasonable and will need to be addressed.
But getting the interface changes using by Green apps right it's not easy or fast. We must make sure those work and have acceptable performances with all supported apps and platform. Such process will involve engineers, testers and designers and will take a rather long period. Therefore the plan is to postpone interface decisions and review and merge these SPV cross validation changes even though they might be unusable or hard to use from Green sessions.

[LeoComandini]

DEBUG=1 ./launch_integration_tests.sh bitcoin
DEBUG=1 ./launch_integration_tests.sh bitcoin

both fails with

thread 'bitcoin' panicked at 'assertion failed: `(left == right)`
  left: `"unconfirmed"`,
 right: `"in_progress"`', tests/test_session.rs:451:9

I guess they need updating

[LeoComandini]

Please add to .gitlab-ci.yml

DEBUG=1 ./launch_integration_tests.sh spv_cross_validate
DEBUG=1 ./launch_integration_tests.sh spv_cross_validation_session

[shesek]

I guess they need updating

Indeed, fixed in d903e0b.

[shesek]

Please add to .gitlab-ci.yml

Added in 31ebd84.

[LeoComandini]

nit: typo "rust-elemenets" in last commit

[LeoComandini]

Also, is this still WIP?

[LeoComandini]

re-ACK c73f427, review diff since last acked version, gitlab CI passed, manually tested that store data (settings and memos) are preserved after upgrade (0.0.40 to c73f427) and downgrade.

[LeoComandini]

Thanks @shesek!