Failed to receive a message from Overseer: Signal channel is terminated and empty.

[ioannist]

Moonbeam-skylake 0.33, operating as a full-node

I am running a fullnode and querying it extensively in localhost. After approx 70 blocks, the moonbeam node crashes with:

Oct 28 05:27:55 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:55 [Relaychain] ✨ Imported #17915340 (0x8d94…96f5)
Oct 28 05:27:55 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:55 [Relaychain] 💤 Idle (6 peers), best: #17915340 (0x8d94…96f5), finalized #17915337 (0x2545…00b5), ⬇ 30.2kiB/s ⬆ 7.0kiB/s
Oct 28 05:27:55 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:55 [🌗] ⚙️  Preparing  0.0 bps, target=#4743523 (11 peers), best: #4743508 (0x95c1…3d43), finalized #4743505 (0x7bff…d019), ⬇ 4.9kiB/s ⬆ 89 B/s
Oct 28 05:27:56 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:56 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0x5adec8fe76ac16a0e2ff5bb1333dab8d683b67ab6fbda537c577511b3d8c511b
Oct 28 05:27:56 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:56 [Relaychain] Failed to fetch runtime API data for job err=NotSupported { runtime_api_name: "validator_groups" }
Oct 28 05:27:56 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:56 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0x5adec8fe76ac16a0e2ff5bb1333dab8d683b67ab6fbda537c577511b3d8c511b
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] Failed to receive a message from Overseer, exiting err=Generated(Context("Signal channel is terminated and empty."))
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] err=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] error receiving message from subsystem context: Generated(Context("Signal channel is terminated and empty.")) err=Generated(Context("Signal channel is terminated and empty."))
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] subsystem exited with error subsystem="statement-distribution-subsystem" err=FromOrigin { origin: "statement-distribution", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] subsystem exited with error subsystem="network-bridge-rx-subsystem" err=FromOrigin { origin: "network-bridge", source: SubsystemError(Generated(Context("Signal channel is terminated and empty."))) }
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] subsystem exited with error subsystem="dispute-distribution-subsystem" err=FromOrigin { origin: "dispute-distribution", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] subsystem exited with error subsystem="availability-recovery-subsystem" err=FromOrigin { origin: "availability-recovery", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] subsystem exited with error subsystem="bitfield-signing-subsystem" err=FromOrigin { origin: "bitfield-signing", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] subsystem exited with error subsystem="candidate-validation-subsystem" err=FromOrigin { origin: "candidate-validation", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] subsystem exited with error subsystem="provisioner-subsystem" err=FromOrigin { origin: "provisioner", source: OverseerExited(Generated(Context("Signal channel is terminated and empty."))) }
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] subsystem exited with error subsystem="network-bridge-tx-subsystem" err=FromOrigin { origin: "network-bridge", source: SubsystemError(Generated(Context("Signal channel is terminated and empty."))) }
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] subsystem exited with error subsystem="chain-api-subsystem" err=FromOrigin { origin: "chain-api", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] Overseer exited with error err=Generated(SubsystemStalled("approval-distribution-subsystem"))
Oct 28 05:27:58 stakebaby-chalandri moonbeam[2508374]: 2023-10-28 05:27:58 [Relaychain] Essential task `overseer` failed. Shutting down service.

This appears to be heavy-load or concurrency related, because the node does not crash if I ease down on the query rate. To put this in context, the node is queried by 7-12 NodeJS processes, each one of which can execute up to 300 queries concurrently. The moonbeam process averages 200%-450% of logical core capacity. I have tried different block spans, and the error persists, so I don't think it's related to db corruption.

Looks like it's a polkadot issue, but I am not sure if it has been resolved or ignored.
paritytech/polkadot#6624

[crystalin]

Thank you @ioannist , We are aware of this issue but couldn't well reproduce it so maybe with your help we can pin point where it comes from

[ioannist]

It eventually happens (within 50 blocks give or take) under heavy load. It does not happen if I only run one indexing worker (or one block at a time).

I've been paying around with flags and setting this, seems to avert the issue
--max-runtime-instances 256

[ioannist]

looks like they are getting to the bottom of it here
paritytech/polkadot-sdk#840

our full node keeps crashing every 20 min or so on this error