Node is not syncing giving peer issue #3058

mayank-daga · 2024-04-15T06:10:11Z

Hi I have archival dockized atom node running. It was running fine until yesterday afternoon and suddenly started failing giving following error in logs :

4:25AM ERR Stopping peer for error err="pong timeout" module=p2p peer="Peer{MConn{168.119.138.215:26656} aa61bc0e8a42eda6ac1276c4279941714a4a38f4 out}"
,4:25AM INF service stop impl=MConn{168.119.138.215:26656} module=p2p msg="Stopping MConnection service" peer=aa61bc0e8a42eda6ac1276c4279941714a4a38f4@168.119.138.215:26656
,4:25AM ERR Connection failed @ sendRoutine conn=MConn{168.119.138.215:26656} err="pong timeout" module=p2p peer=aa61bc0e8a42eda6ac1276c4279941714a4a38f4@168.119.138.215:26656
,4:25AM INF service stop impl="Peer{MConn{168.119.138.215:26656} aa61bc0e8a42eda6ac1276c4279941714a4a38f4 out}" module=p2p msg="Stopping Peer service" peer=aa61bc0e8a42eda6ac1276c4279941714a4a38f4@168.119.138.215:26656
,4:25AM INF Reconnecting to peer addr=aa61bc0e8a42eda6ac1276c4279941714a4a38f4@168.119.138.215:26656 module=p2p
,4:25AM INF service start impl="Peer{MConn{168.119.138.215:26656} aa61bc0e8a42eda6ac1276c4279941714a4a38f4 out}" module=p2p msg="Starting Peer service" peer=aa61bc0e8a42eda6ac1276c4279941714a4a38f4@168.119.138.215:26656
,4:25AM INF service start impl=MConn{168.119.138.215:26656} module=p2p msg="Starting MConnection service" peer=aa61bc0e8a42eda6ac1276c4279941714a4a38f4@168.119.138.215:26656
,4:25AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{84.46.248.91:26660} module=p2p peer=b459d7c9e1193af80e7fba58f2b8b0008be1e619@84.46.248.91:26660
,4:25AM INF service stop impl=MConn{84.46.248.91:26660} module=p2p msg="Stopping MConnection service" peer=b459d7c9e1193af80e7fba58f2b8b0008be1e619@84.46.248.91:26660
,4:25AM INF Reconnecting to peer addr=b459d7c9e1193af80e7fba58f2b8b0008be1e619@84.46.248.91:26660 module=p2p
,4:25AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{84.46.248.91:26660} b459d7c9e1193af80e7fba58f2b8b0008be1e619 out}"
,4:25AM INF service stop impl="Peer{MConn{84.46.248.91:26660} b459d7c9e1193af80e7fba58f2b8b0008be1e619 out}" module=p2p msg="Stopping Peer service" peer=b459d7c9e1193af80e7fba58f2b8b0008be1e619@84.46.248.91:26660
,4:25AM INF service start impl="Peer{MConn{84.46.248.91:26660} b459d7c9e1193af80e7fba58f2b8b0008be1e619 out}" module=p2p msg="Starting Peer service" peer=b459d7c9e1193af80e7fba58f2b8b0008be1e619@84.46.248.91:26660
,4:25AM INF service start impl=MConn{84.46.248.91:26660} module=p2p msg="Starting MConnection service" peer=b459d7c9e1193af80e7fba58f2b8b0008be1e619@84.46.248.91:26660

I stopped and restarted the node but same issue.
My Node version is 15.1.0

MSalopek · 2024-04-15T07:24:48Z

@mayank-daga

I stopped and restarted the node but same issue.
My Node version is 15.1.0

Please take note that the mainnet is running v15.2.0. You seem to be running v15.1.0.
The error messages seem to be the node just looking for peers. Please check your logs for a apphash error logs.

Please check your error logs for apphash errors.
Please try updating the node to v15.2.0 and try again.

mayank-daga · 2024-04-15T07:48:51Z

HI @MSalopek i searched with keyword apphash in logs and found below logs:

7:47AM ERR Stopping peer for error err="Reactor validation error: wrong Block.Header.AppHash. Expected 91A61EF62726DC6454C7E825A59E9E50887699D37F97E5FF9613550C7369859C, got 673D16879AE743BA4BF000E2F488D3A570D3F9B12FC314EC61039701868A84A9" module=p2p peer="Peer{MConn{198.244.202.140:26656} e829d4764a5cecc44b3414777853b34407b36601 out}"

MSalopek · 2024-04-15T08:14:48Z

Your node apphashed because you were running v15.1.0. v15.2.0 and v15.1.0 have differences in how they process data.

A coordinated upgrade happened at height 19939000, gaia v15.1.0 was upgraded to v15.2.0.

If you have an earlier snapshot or a backup you can restart with gaiad v15.2.0 and let the node catch up.

Alternatively, depending on your infrastructure, you may attempt to rollback your gaia node (the tool can rollback 1 block), then you can restart with v15.2.0 binary.

# rollback a single block
gaiad rollback

# re-start with new binary
gaiad-v15.2.0 start

Re-starting from a snapshot or a backup is the best option here.

mayank-daga · 2024-04-15T09:11:52Z

Hi @MSalopek getting below error logs when using 15.2.0 images:

8:13AM INF Replay: Vote blockID=:0:000000000000 height=19971037 module=consensus peer=a9d9fa57f1b059a1b1ef1759f5aea02c059e82b7 round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=E181A2570FCE5604DE1926DFF0A2F40E6DE4E21F36D8CAB9E01B270EF4862531:2:F54153137AE1 height=19971037 module=consensus peer=a9d9fa57f1b059a1b1ef1759f5aea02c059e82b7 round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=:0:000000000000 height=19971037 module=consensus peer=a9d9fa57f1b059a1b1ef1759f5aea02c059e82b7 round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=E181A2570FCE5604DE1926DFF0A2F40E6DE4E21F36D8CAB9E01B270EF4862531:2:F54153137AE1 height=19971037 module=consensus peer=a9d9fa57f1b059a1b1ef1759f5aea02c059e82b7 round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=:0:000000000000 height=19971037 module=consensus peer=a9d9fa57f1b059a1b1ef1759f5aea02c059e82b7 round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=E181A2570FCE5604DE1926DFF0A2F40E6DE4E21F36D8CAB9E01B270EF4862531:2:F54153137AE1 height=19971037 module=consensus peer=a9d9fa57f1b059a1b1ef1759f5aea02c059e82b7 round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=:0:000000000000 height=19971037 module=consensus peer=a9d9fa57f1b059a1b1ef1759f5aea02c059e82b7 round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=E181A2570FCE5604DE1926DFF0A2F40E6DE4E21F36D8CAB9E01B270EF4862531:2:F54153137AE1 height=19971037 module=consensus peer=a9d9fa57f1b059a1b1ef1759f5aea02c059e82b7 round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=:0:000000000000 height=19971037 module=consensus peer=a9d9fa57f1b059a1b1ef1759f5aea02c059e82b7 round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=:0:000000000000 height=19971037 module=consensus peer=10e4f3a4b4f8bc0168933f2daeb379b84354d5ad round=0 type=SIGNED_MSG_TYPE_PREVOTE

8:13AM INF Replay: Vote blockID=E181A2570FCE5604DE1926DFF0A2F40E6DE4E21F36D8CAB9E01B270EF4862531:2:F54153137AE1 height=19971037 module=consensus peer=10e4f3a4b4f8bc0168933f2daeb379b84354d5ad round=0 type=SIGNED_MSG_TYPE_PREVOTE