Run full sync tests on Mainnet

[teor2345]

Motivation

We want to avoid merging PRs if their changes break syncing to the tip.

Test Plan

Run a mainnet sync, and make sure they reach block X within N hours:

• N is the typical time on that infrastructure, plus a small margin
• X is the estimated block height, calculated using a recent height and the target block interval (implemented in Full chain synchronization acceptance tests #3543)

We should also check the logs to make sure Zebra stops syncing blocks, and activates the mempool.
(Implemented in #3543 and #3555.)

We need to run a full sync, and that takes a few hours, so we should do it on Google Cloud. (If we run a short sync, we won't catch slowness or hang issues with post-checkpoint syncs.)

Follow Up Work

TODO: create tickets for leftover work

As a follow-up, we could show sync speeds over time. This history will help us diagnose issues more reliably.

As another follow-up, we should run these tests on testnet. To run a reliable testnet sync, we need to deploy more testnet Zebra or zcashd instances, and make sure they appear in DNS seeders (#2380, #1904). So that's out of scope for now.

[teor2345]

I've been using these command-lines to test #3543 and #3555:

FULL_SYNC_MAINNET_TIMEOUT_MINUTES=600 cargo test --release --test acceptance -- --nocapture --ignored full_sync_mainnet
FULL_SYNC_TESTNET_TIMEOUT_MINUTES=600 cargo test --release --test acceptance -- --nocapture --ignored full_sync_testnet

https://doc.rust-lang.org/rustc/tests/index.html#--nocapture

[jvff]

I'm not sure if this should be a new job in .github/workflows/test.yml or if I should change the existing test-stateful-sync job. @gustavovalverde, what do you think?

[teor2345]

I'm not sure if this should be a new job in .github/workflows/test.yml or if I should change the existing test-stateful-sync job.

I'd like to keep test-stateful-sync, and make it a required job. It's quick, and it tells us if we've broken full validation on the first thousand post-canopy blocks.

(So it catches most full validation bugs.)

[gustavovalverde]

Fixed by #3582