Relay chain subsytem error

[bensacc]

Attempting to spin up a new Moonriver node using purestake/moonbeam:v0.30.0, I have uit running in a docker container and after several hours of seemingly successfull syncing I'm receiving these errors:

2023-03-20 22:25:08 [Relaychain] ⚙️ Syncing 153.6 bps, target=#17126217 (40 peers), best: #7143039 (0xfc86…89fd), finalized #7142912 (0xa888…2121), ⬇ 519.2kiB/s ⬆ 247.2kiB/s
2023-03-20 22:25:12 [Relaychain] subsystem exited with error subsystem="candidate-validation-subsystem" err=FromOrigin { origin: "candidate-validation", source: Generated(Context("Signal channel is terminated and empty.")) }
2023-03-20 22:25:12 [Relaychain] subsystem exited with error subsystem="approval-voting-subsystem" err=FromOrigin { origin: "approval-voting", source: Generated(Context("Signal channel is terminated and empty.")) }
2023-03-20 22:25:12 [Relaychain] subsystem exited with error subsystem="chain-api-subsystem" err=FromOrigin { origin: "chain-api", source: Generated(Context("Signal channel is terminated and empty.")) }
2023-03-20 22:25:12 [Relaychain] subsystem exited with error subsystem="runtime-api-subsystem" err=Generated(Context("Signal channel is terminated and empty."))
2023-03-20 22:25:12 [Relaychain] subsystem exited with error subsystem="bitfield-signing-subsystem" err=FromOrigin { origin: "bitfield-signing", source: Generated(Context("Signal channel is terminated and empty.")) }
2023-03-20 22:25:12 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-03-20 22:25:12 [Relaychain] subsystem exited with error subsystem="statement-distribution-subsystem" err=FromOrigin { origin: "statement-distribution", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }
2023-03-20 22:25:12 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-03-20 22:25:12 [Relaychain] subsystem exited with error subsystem="availability-recovery-subsystem" err=FromOrigin { origin: "availability-recovery", source: Generated(Context("Signal channel is terminated and empty.")) }
2023-03-20 22:25:12 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-03-20 22:25:12 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-03-20 22:25:12 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-03-20 22:25:12 [Relaychain] error receiving message from subsystem context: Generated(Context("Signal channel is terminated and empty.")) err=Generated(Context("Signal channel is terminated and empty."))
2023-03-20 22:25:12 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))

after which "[Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))" is repeated until I kill it

docker container config:

docker run --restart=always -p 9944 -p 9944:9944 -p 30333 -p 30333:30333 -d -v "/var/lib/moonriver-data:/data" -u $(id -u ${USER}):$(id -g ${USER}) --name="moonriver-primary" purestake/moonbeam:v0.30.0 --base-path=/data --chain moonriver --name="moonriver-primary" --execution wasm --wasm-execution compiled --state-pruning archive --trie-cache-size 0 --ws-port 9944 --ws-max-connections 100 --db-cache 16000 -- --name="moonriver-primary (Embedded Relay)

(note: "-id -u ${USER}", is just getting formatted weird in the text above)

anyone have any idea what could be causing this?? thanks so much in advance...

[crystalin]

@bensacc is it still appearing after restarting ?

[bensacc]

@crystalin It is, same as the original occurrence, after a few hours

[bensacc]

Maybe I'm a bit confused as to whether I should be running port 9944 or 9945. I was using 9944, but have tried switching to 9945, and so far it's continuing to sync, I guess we'll see in an hour or so.

[bensacc]

I see a number of these in the logs as well, following the restart, but maybe unrelated:

cannot query the runtime API version: UnknownBlock: State already discarded for 0x64bbaf57336d698c00eaff6d6847631ca47ba797276590b8033467f1780daebb
(actually hash varies, of course)

they don't seem to be causing any larger issues

[crystalin]

@bkchr this is what I mentioned before, we also experienced this issue (not the "Signal channel is terminated and empty" but the "State already discarded for") few times on our side :/

[bkchr]

State already discarded for

Should be fixed by: paritytech/polkadot-sdk#793

For the other issue, please report here: paritytech/polkadot#6759

[davidrhodus]

Seeing similar issue here. Unable to perform full sync

./moonbeam --version
moonbeam 0.30.3-7f85ea4d5f6

2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
2023-04-08 17:30:22 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))

[Szymongib]

It seems like it is still the problem... I cannot sync new node even from a snapshot. It syncs for an hour or so and then this happens, eventually crashing the container. Running v0.31.1.

Last logs, before it continues to spam the same line:

moonbeam_1  | 2023-05-22 19:50:07 [🌗] ✨ Imported #3618819 (0x78d7…fee6)
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] ⚙️  Syncing 506.2 bps, target=#15637045 (40 peers), best: #2727953 (0x722b…268b), finalized #2727936 (0xbb51…4fce), ⬇ 517.5kiB/s ⬆ 294.1kiB/s
moonbeam_1  | 2023-05-22 19:50:10 [🌗] 💤 Idle (21 peers), best: #3618673 (0xbf5e…d874), finalized #3617473 (0x21fc…6191), ⬇ 12.7kiB/s ⬆ 16.9kiB/s
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xe9536cd2fe5b4d998e4286c1dae731bfafdd94dabcd56c2cc20b56ef93637d01
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="availability-recovery-subsystem" err=FromOrigin { origin: "availability-recovery", source: Generated(Context("Signal channel is terminated and empty.")) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="statement-distribution-subsystem" err=FromOrigin { origin: "statement-distribution", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="approval-voting-subsystem" err=FromOrigin { origin: "approval-voting", source: Generated(Context("Signal channel is terminated and empty.")) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="runtime-api-subsystem" err=Generated(Context("Signal channel is terminated and empty."))
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="candidate-validation-subsystem" err=FromOrigin { origin: "candidate-validation", source: Generated(Context("Signal channel is terminated and empty.")) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="availability-distribution-subsystem" err=FromOrigin { origin: "availability-distribution", source: IncomingMessageChannel(Generated(Context("Signal channel is terminated and empty."))) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="chain-api-subsystem" err=FromOrigin { origin: "chain-api", source: Generated(Context("Signal channel is terminated and empty.")) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] Overseer exited with error err=Generated(SubsystemStalled("availability-store-subsystem"))
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="bitfield-signing-subsystem" err=FromOrigin { origin: "bitfield-signing", source: Generated(Context("Signal channel is terminated and empty.")) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] Failed to receive a message from Overseer, exiting err=Generated(Context("Signal channel is terminated and empty."))
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] error receiving message from subsystem context: Generated(Context("Signal channel is terminated and empty.")) err=Generated(Context("Signal channel is terminated and empty."))
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="network-bridge-rx-subsystem" err=FromOrigin { origin: "network-bridge", source: SubsystemError(Generated(Context("Signal channel is terminated and empty."))) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="network-bridge-tx-subsystem" err=FromOrigin { origin: "network-bridge", source: SubsystemError(Generated(Context("Signal channel is terminated and empty."))) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] subsystem exited with error subsystem="dispute-distribution-subsystem" err=FromOrigin { origin: "dispute-distribution", source: SubsystemReceive(Generated(Context("Signal channel is terminated and empty."))) }
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] err=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
moonbeam_1  | 2023-05-22 19:50:10 [Relaychain] error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))

[crystalin]

@Szymongib We are still investigating the sync issue. The error message is less problematic

[chengjiao]

what I am getting from log is,

2023-05-25 09:43:27 [Relaychain] ⚙️  Syncing 204.8 bps, target=#18064754 (40 peers), best: #5136896 (0x0113…2030), finalized #5136384 (0x4190…993c), ⬇ 320.9kiB/s ⬆ 122.7kiB/s    
2023-05-25 09:43:27 [🌗] ⚙️  Syncing 20.6 bps, target=#4314303 (16 peers), best: #353867 (0xfbcc…d088), finalized #0 (0x401a…474b), ⬇ 3.2kiB/s ⬆ 2.3kiB/s    
2023-05-25 09:43:29 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xa92811003344380248e15013c691721f20f13c39f4e57ac7dbc464549590e6b7
2023-05-25 09:43:31 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xa92811003344380248e15013c691721f20f13c39f4e57ac7dbc464549590e6b7
2023-05-25 09:43:32 [Relaychain] ⚙️  Syncing 168.0 bps, target=#18064754 (40 peers), best: #5137737 (0x508d…f97b), finalized #5137408 (0x2c18…028d), ⬇ 300.4kiB/s ⬆ 113.1kiB/s    
2023-05-25 09:43:32 [🌗] ⚙️  Syncing 21.0 bps, target=#4314303 (16 peers), best: #353972 (0xad8f…0c16), finalized #0 (0x401a…474b), ⬇ 106.6kiB/s ⬆ 1.6kiB/s    
2023-05-25 09:43:35 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xa92811003344380248e15013c691721f20f13c39f4e57ac7dbc464549590e6b7
2023-05-25 09:43:37 [Relaychain] ⚙️  Syncing 187.2 bps, target=#18064754 (40 peers), best: #5138673 (0x4e63…d7d5), finalized #5138469 (0x2062…6149), ⬇ 260.7kiB/s ⬆ 104.6kiB/s    
2023-05-25 09:43:37 [🌗] ⚙️  Syncing 21.0 bps, target=#4314303 (16 peers), best: #354077 (0x01f4…1d57), finalized #0 (0x401a…474b), ⬇ 69.6kiB/s ⬆ 0.9kiB/s    
2023-05-25 09:43:37 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xa92811003344380248e15013c691721f20f13c39f4e57ac7dbc464549590e6b7
2023-05-25 09:43:39 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xa92811003344380248e15013c691721f20f13c39f4e57ac7dbc464549590e6b7
2023-05-25 09:43:41 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xa92811003344380248e15013c691721f20f13c39f4e57ac7dbc464549590e6b7
2023-05-25 09:43:42 [Relaychain] ⚙️  Syncing 226.0 bps, target=#18064754 (40 peers), best: #5139803 (0xf41f…ae52), finalized #5139456 (0x7bb0…9bcc), ⬇ 343.9kiB/s ⬆ 121.1kiB/s    
2023-05-25 09:43:42 [🌗] ⚙️  Syncing 19.8 bps, target=#4314303 (16 peers), best: #354176 (0xc051…d258), finalized #0 (0x401a…474b), ⬇ 2.2kiB/s ⬆ 2.3kiB/s    
2023-05-25 09:43:43 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xa92811003344380248e15013c691721f20f13c39f4e57ac7dbc464549590e6b7
2023-05-25 09:43:45 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xa92811003344380248e15013c691721f20f13c39f4e57ac7dbc464549590e6b7
2023-05-25 09:43:47 [Relaychain] ⚙️  Syncing 237.8 bps, target=#18064754 (40 peers), best: #5140992 (0x450c…c4a6), finalized #5139968 (0xf252…76ca), ⬇ 377.9kiB/s ⬆ 111.6kiB/s    
2023-05-25 09:43:47 [🌗] ⚙️  Syncing 20.4 bps, target=#4314303 (16 peers), best: #354278 (0x7f25…6b9f), finalized #0 (0x401a…474b), ⬇ 108.0kiB/s ⬆ 1.6kiB/s    
2023-05-25 09:43:49 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0xa92811003344380248e15013c691721f20f13c39f4e57ac7dbc464549590e6b7

@Szymongib where is the snapshot you use? is it public for usage? I want to have a try

[Szymongib]

@chengjiao I have used p2p snapshot https://p2p.org/snapshot/moonbeam

What ended up working for me was setting up both Moonbeam and Polkadot data from snapshot.