MBA - New install stops syncing on block 1131146

[JB273]

Installed a new server with Ubuntu 22.04 to setup a new MBA Archive RPC node.
It starts of fine, but after a while it stalls syncing blocks, currently it stalled at block 1131146.
Restarting the service gives the following issues in the log:
Jun 17 13:43:16 rpc05-var moonbase[50064]: 2023-06-17 13:43:16 [🌗] panicked at 'Digest item must match that calculated.', /home/builder/cargo/git/checkouts/substrate-189071a041b0d328/ecd04ec/frame/executive/src/lib.rs:668:13 Jun 17 13:43:16 rpc05-var moonbase[50064]: 2023-06-17 13:43:16 [🌗] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm unreachableinstruction executed Jun 17 13:43:16 rpc05-var moonbase[50064]: WASM backtrace: Jun 17 13:43:16 rpc05-var moonbase[50064]: error while executing at wasm backtrace: Jun 17 13:43:16 rpc05-var moonbase[50064]: 0: 0x3c3c - <unknown>!rust_begin_unwind Jun 17 13:43:16 rpc05-var moonbase[50064]: 1: 0x2980 - <unknown>!core::panicking::panic_fmt::h18b15be283411c65 Jun 17 13:43:16 rpc05-var moonbase[50064]: 2: 0x3bbd - <unknown>!core::panicking::panic::h0207b244811fef8b Jun 17 13:43:16 rpc05-var moonbase[50064]: 3: 0x10144c - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h90a2a2b5f2ce55e4 Jun 17 13:43:16 rpc05-var moonbase[50064]: 4: 0x2c4c85 - <unknown>!Core_execute_block Jun 17 13:43:16 rpc05-var moonbase[50064]: 2023-06-17 13:43:16 [🌗] 💔 Error importing block 0x236e62a41257327fed36be064504a0f765169c0e8c50328033394f6b5ae45c16: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasmunreachableinstruction executed Jun 17 13:43:16 rpc05-var moonbase[50064]: WASM backtrace: Jun 17 13:43:16 rpc05-var moonbase[50064]: error while executing at wasm backtrace: Jun 17 13:43:16 rpc05-var moonbase[50064]: 0: 0x3c3c - <unknown>!rust_begin_unwind Jun 17 13:43:16 rpc05-var moonbase[50064]: 1: 0x2980 - <unknown>!core::panicking::panic_fmt::h18b15be283411c65 Jun 17 13:43:16 rpc05-var moonbase[50064]: 2: 0x3bbd - <unknown>!core::panicking::panic::h0207b244811fef8b Jun 17 13:43:16 rpc05-var moonbase[50064]: 3: 0x10144c - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h90a2a2b5f2ce55e4 Jun 17 13:43:16 rpc05-var moonbase[50064]: 4: 0x2c4c85 - <unknown>!Core_execute_block
and
Jun 17 13:43:18 rpc05-var moonbase[50064]: 2023-06-17 13:43:18 [Relaychain] Skipping chain: unable to gather info about imported block (0xbcebf6cdd7959a4ef3db86c8abfbaf4aea0ff1bcb4571106691d11d97ac33e52, 10782350): session info unavailable Jun 17 13:43:18 rpc05-var moonbase[50064]: 2023-06-17 13:43:18 [Relaychain] Could not retrieve session info from rolling session window session=18042 Jun 17 13:43:18 rpc05-var moonbase[50064]: 2023-06-17 13:43:18 [Relaychain] Could not retrieve session info from rolling session window session=18042 Jun 17 13:43:18 rpc05-var moonbase[50064]: 2023-06-17 13:43:18 [Relaychain] Could not retrieve session info from rolling session window session=18042 Jun 17 13:43:19 rpc05-var moonbase[50064]: 2023-06-17 13:43:19 [Relaychain] 💤 Idle (4 peers), best: #10782352 (0x4d32…eb2c), finalized #10782349 (0x4375…d5d9), ⬇ 160.4kiB/s ⬆ 79.8kiB/s Jun 17 13:43:19 rpc05-var moonbase[50064]: 2023-06-17 13:43:19 [🌗] ⚙️ Syncing, target=#4559211 (1 peers), best: #1131146 (0xe3f7…8c36), finalized #3497 (0x6331…793a), ⬇ 5.0MiB/s ⬆ 38.5kiB/s Jun 17 13:43:21 rpc05-var moonbase[50064]: 2023-06-17 13:43:21 [🌗] panicked at 'Digest item must match that calculated.', /home/builder/cargo/git/checkouts/substrate-189071a041b0d328/ecd04ec/frame/executive/src/lib.rs:668:13 Jun 17 13:43:21 rpc05-var moonbase[50064]: 2023-06-17 13:43:21 [🌗] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm unreachableinstruction executed Jun 17 13:43:21 rpc05-var moonbase[50064]: WASM backtrace: Jun 17 13:43:21 rpc05-var moonbase[50064]: error while executing at wasm backtrace: Jun 17 13:43:21 rpc05-var moonbase[50064]: 0: 0x3c3c - <unknown>!rust_begin_unwind Jun 17 13:43:21 rpc05-var moonbase[50064]: 1: 0x2980 - <unknown>!core::panicking::panic_fmt::h18b15be283411c65 Jun 17 13:43:21 rpc05-var moonbase[50064]: 2: 0x3bbd - <unknown>!core::panicking::panic::h0207b244811fef8b Jun 17 13:43:21 rpc05-var moonbase[50064]: 3: 0x10144c - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h90a2a2b5f2ce55e4 Jun 17 13:43:21 rpc05-var moonbase[50064]: 4: 0x2c4c85 - <unknown>!Core_execute_block Jun 17 13:43:21 rpc05-var moonbase[50064]: 2023-06-17 13:43:21 [🌗] 💔 Error importing block 0x236e62a41257327fed36be064504a0f765169c0e8c50328033394f6b5ae45c16: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasmunreachableinstruction executed Jun 17 13:43:21 rpc05-var moonbase[50064]: WASM backtrace: Jun 17 13:43:21 rpc05-var moonbase[50064]: error while executing at wasm backtrace: Jun 17 13:43:21 rpc05-var moonbase[50064]: 0: 0x3c3c - <unknown>!rust_begin_unwind Jun 17 13:43:21 rpc05-var moonbase[50064]: 1: 0x2980 - <unknown>!core::panicking::panic_fmt::h18b15be283411c65 Jun 17 13:43:21 rpc05-var moonbase[50064]: 2: 0x3bbd - <unknown>!core::panicking::panic::h0207b244811fef8b Jun 17 13:43:21 rpc05-var moonbase[50064]: 3: 0x10144c - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h90a2a2b5f2ce55e4 Jun 17 13:43:21 rpc05-var moonbase[50064]: 4: 0x2c4c85 - <unknown>!Core_execute_block

[JB273]

Upgraded binary to v0.32.1, started a clean sync (Archival) and is stalls on the same block number. (1131146)

[crystalin]

Thank you, we are taking a look at it now

[JB273]

ExecStart=/home/pops/moonbase/moonbase \
     --port 30335 \
     --ws-port 9946 \
     --ws-max-connections 10000 \
     --execution wasm \
     --wasm-execution compiled \
     --state-pruning=archive \
     --prometheus-external \
     --prometheus-port 9617 \
     --trie-cache-size 4 \
     --db-cache 32000 \
     --base-path /home/pops/moonbase \
     --ethapi=debug,trace,txpool \
     --wasm-runtime-overrides=/home/pops/moonbase/wasm \
     --chain alphanet \
     --name "P-OPS Team Moonbase RPC-2" \
     --runtime-cache-size 64 \
     --rpc-max-response-size 128 \
     --unsafe-rpc-external \
     --unsafe-ws-external \
     --rpc-cors all \
     -- \
     --port 30336 \
     --ws-port 9947 \
     --execution wasm \
     --pruning=1000 \
     --prometheus-external \
     --prometheus-port 9618 \
     --name="P-OPS Team Moonbase RPC-2 (Embedded Relay)"

[crystalin]

Multiple engineers, including me have been trying to reproduce this, but none of us got issue synchronizing past 1131147.
I'm not sure what happens in your case, but I suspect that the node might have been turned off in a middle of a DB process leaving data unstable when you restarted it, leading to the node not able to process that block anymore.

Can you resync from scratch, using client v0.32.1 to see if this happens again ?

[JB273]

Resynced from scratch... still not passing block 1131146, not sure why this is... It's also odd as on the same machine Moonbeam synced flawless!