GETH Node Timing out and not syncing

[Sreeram1993]

Beacon and GETH nodes not syncing as expected

I am having an ETH archival node and a beacon node. Both have been syncing for few months and have also been upgraded for the recent Duncan uprade as well. It have been syncing post upgrade too but for last 4-5 days the performance have degraded and the geth node is 16hours behind. Beacon also shows warnings.

--- BEACON NODE ---
time="2024-04-05 05:59:26" level=warning msg="Skip processing batched blocks" error="beacon node doesn't have a parent in db with root: 0x8da6933cb7c984d8c7e0503b98b88043db2e1f8cfcc479f02066af621e4a6135 (in processBatchedBlocks, slot=8784576)" prefix=initial-sync
time="2024-04-05 05:59:31" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 05:59:45" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 05:59:59" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:00:11" level=info msg="Peer summary" activePeers=61 inbound=0 outbound=61 prefix=p2p
time="2024-04-05 06:00:13" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:00:27" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:00:41" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:00:55" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:01:09" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:01:11" level=info msg="Peer summary" activePeers=61 inbound=0 outbound=61 prefix=p2p
time="2024-04-05 06:01:23" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:01:37" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:01:51" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:02:05" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:02:11" level=info msg="Peer summary" activePeers=61 inbound=0 outbound=60 prefix=p2p
time="2024-04-05 06:02:19" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:02:33" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:02:47" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:03:01" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:03:11" level=info msg="Peer summary" activePeers=58 inbound=0 outbound=57 prefix=p2p
time="2024-04-05 06:03:15" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:03:29" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:03:43" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:03:51" level=info msg="Processing block batch of size 64 starting from 0x09e37c18... 8784448/8789417 - estimated time remaining 25m52s" blocksPerSecond=3.2 peers=62 prefix=initial-sync
time="2024-04-05 06:03:57" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:04:11" level=info msg="Peer summary" activePeers=66 inbound=0 outbound=53 prefix=p2p
time="2024-04-05 06:04:11" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:04:16" level=warning msg="Skip processing batched blocks" error="could not process block in batch: timeout from http.Client: received an undefined execution engine error" prefix=initial-sync
time="2024-04-05 06:04:25" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:04:39" level=warning msg="Execution client is not syncing" prefix=execution
time="2024-04-05 06:04:53" level=warning msg="Execution client is not syncing" prefix=execution

--- GETH NODE ---
WARN [04-05|06:05:14.347] Ignoring already known beacon payload number=19,582,732 hash=797ba2..2b275e age=16h30m3s
WARN [04-05|06:05:14.355] Ignoring already known beacon payload number=19,582,733 hash=2ff297..555304 age=16h29m51s
WARN [04-05|06:05:14.362] Ignoring already known beacon payload number=19,582,734 hash=e59583..13d889 age=16h29m39s
WARN [04-05|06:05:14.370] Ignoring already known beacon payload number=19,582,735 hash=8c8243..281e42 age=16h29m27s
WARN [04-05|06:05:14.389] Ignoring already known beacon payload number=19,582,736 hash=5aaf86..1f94e3 age=16h29m15s
WARN [04-05|06:05:14.398] Ignoring already known beacon payload number=19,582,737 hash=98f39e..d20658 age=16h29m3s
WARN [04-05|06:05:14.437] Ignoring already known beacon payload number=19,582,738 hash=15719a..4626dc age=16h28m51s
WARN [04-05|06:05:14.765] Ignoring already known beacon payload number=19,582,739 hash=f33ba1..5d7729 age=16h28m39s
WARN [04-05|06:05:14.787] Ignoring already known beacon payload number=19,582,740 hash=b76873..c0aaba age=16h28m27s
WARN [04-05|06:05:14.807] Ignoring already known beacon payload number=19,582,741 hash=ef2660..2587e4 age=16h28m15s
WARN [04-05|06:05:14.828] Ignoring already known beacon payload number=19,582,742 hash=584b08..3e8baa age=16h28m3s
WARN [04-05|06:05:14.847] Ignoring already known beacon payload number=19,582,743 hash=0dffce..676190 age=16h27m51s
WARN [04-05|06:05:14.864] Ignoring already known beacon payload number=19,582,744 hash=8f59d9..de171e age=16h27m39s
WARN [04-05|06:05:14.897] Ignoring already known beacon payload number=19,582,745 hash=33f41d..e53fe7 age=16h27m27s
WARN [04-05|06:05:14.927] Ignoring already known beacon payload number=19,582,746 hash=ffdba1..c58bbf age=16h27m15s
WARN [04-05|06:05:14.946] Ignoring already known beacon payload number=19,582,747 hash=262f98..26ee38 age=16h27m3s
WARN [04-05|06:05:14.953] Ignoring already known beacon payload number=19,582,748 hash=ccff4e..8a33a8 age=16h26m51s
WARN [04-05|06:05:14.960] Ignoring already known beacon payload number=19,582,749 hash=e4a508..0d4670 age=16h26m39s
WARN [04-05|06:05:14.978] Ignoring already known beacon payload number=19,582,750 hash=d96513..03850f age=16h26m27s
WARN [04-05|06:05:14.987] Ignoring already known beacon payload number=19,582,751 hash=88e042..b778ae age=16h26m15s
WARN [04-05|06:05:14.998] Ignoring already known beacon payload number=19,582,752 hash=50ab7c..62863c age=16h26m3s
WARN [04-05|06:05:15.017] Ignoring already known beacon payload number=19,582,753 hash=d70d4f..8df50c age=16h25m52s
WARN [04-05|06:05:15.020] Ignoring already known beacon payload number=19,582,754 hash=c4b858..4ca02e age=16h25m40s
WARN [04-05|06:05:15.042] Ignoring already known beacon payload number=19,582,755 hash=6efdce..96b738 age=16h25m28s
WARN [04-05|06:05:15.060] Ignoring already known beacon payload number=19,582,756 hash=e41d4a..c82d79 age=16h25m16s
WARN [04-05|06:05:15.065] Ignoring already known beacon payload number=19,582,757 hash=343047..67f0f5 age=16h25m4s
WARN [04-05|06:05:15.090] Ignoring already known beacon payload number=19,582,758 hash=23c150..895f35 age=16h24m52s
WARN [04-05|06:05:15.099] Ignoring already known beacon payload number=19,582,759 hash=6d55b5..ec839e age=16h24m40s
WARN [04-05|06:05:15.102] Ignoring already known beacon payload number=19,582,760 hash=c9cc56..3d562d age=16h24m28s
WARN [04-05|06:05:15.124] Ignoring already known beacon payload number=19,582,761 hash=9f1db3..8ceffb age=16h24m16s
INFO [04-05|06:05:25.335] Imported new potential chain segment number=19,582,762 hash=837eae..b93dbc blocks=1 txs=66 mgas=4.271 elapsed=10.167s mgasps=0.420 age=16h24m14s triedirty=0.00B

I have redeployed both the nodes using snapshot as well but still it's not syncing and lagging behind. Any idea what could be the issue ?

[MariusVanDerWijden]

INFO [04-05|06:05:25.335] Imported new potential chain segment number=19,582,762 hash=837eae..b93dbc blocks=1 txs=66 mgas=4.271 elapsed=10.167s mgasps=0.420 age=16h24m14s triedirty=0.00B

This means your node is syncing, but its incredibly slow. 10 seconds for 1 block. Are any of your stats (CPU, IO, RAM) maxed out?

[Sreeram1993]

INFO [04-05|06:05:25.335] Imported new potential chain segment number=19,582,762 hash=837eae..b93dbc blocks=1 txs=66 mgas=4.271 elapsed=10.167s mgasps=0.420 age=16h24m14s triedirty=0.00B

This means your node is syncing, but its incredibly slow. 10 seconds for 1 block. Are any of your stats (CPU, IO, RAM) maxed out?

None so far.

[Sreeram1993]

INFO [04-05|06:05:25.335] Imported new potential chain segment number=19,582,762 hash=837eae..b93dbc blocks=1 txs=66 mgas=4.271 elapsed=10.167s mgasps=0.420 age=16h24m14s triedirty=0.00B

This means your node is syncing, but its incredibly slow. 10 seconds for 1 block. Are any of your stats (CPU, IO, RAM) maxed out?

Sharing the args from geth as memory looks good for the nodes.

---

• args:
  - '--cache=32768'
  - '--cache.trie=30'
  - '--cache.gc=0'
  - '--cache.snapshot=20'
  - '--cache.preimages'
  - '--cache.database=70'
  - '--syncmode=full'
  - '--gcmode=archive'
  - '--datadir=/data/eth'
  - '--snapshot=false'
  - '--maxpeers=20'
  - '--http'
  - '--http.api=eth,web3,admin,debug'
  - '--http.addr=0.0.0.0'
  - '--http.corsdomain='
  - '--http.vhosts='
  - '--graphql'
  - '--graphql.vhosts='
  - '--graphql.corsdomain='
  - '--ws'
  - '--ws.addr=0.0.0.0'
  - '--authrpc.addr=0.0.0.0'
  - '--authrpc.jwtsecret=/jwt.hex'
  - '--authrpc.vhosts=*'
  - '--pprof.addr=0.0.0.0'
  - '--metrics.addr=0.0.0.0'

---

[MariusVanDerWijden]

You're running an archive node?

[Sreeram1993]

You're running an archive node?

Yes, this node was running till 5 days back. Suddenly it started deteriorating. So we had another node from which we took the snapshot of the archival disk and redeployed. But that also hasn't changed the sync speed for this node. It's now 18 hours behind.

[MariusVanDerWijden]

Hmm sounds like a disk issue to me, could you do a speedtest/smart scan of your disks?

[Jyoti-singh18]

1073741824 bytes (1.0GB) copied, 10.480924 seconds, 97.7MB/s

This is what we see, any thoughts? @MariusVanDerWijden