Klaytn Node Not Syncing Properly

[mayank-daga]

I have Klay Node Setup in my server and it was syncing but recently i have started getting this log:

INFO[03/27,07:39:33 Z] [33] Added a multichannel P2P Peer id=301c8d7d8f3b0352 conn=trusted-staticdial peerID=301c8d7d8f3b0352
WARN[03/27,07:39:37 Z] [28] Synchronisation failed, dropping peer peer=438ee8575f07e8f8 err=timeout
ERROR[03/27,07:39:37 Z] [33] ProtocolManager failed to handle message id=f060f60df76db002 conn=trusted-staticdial msg="msg #11 (5220 bytes)" err="Invalid message - msg #11 (5220 bytes): invalid message: (code b) (size 5220) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
INFO[03/27,07:39:37 Z] [33] Disconnected a multichannel P2P Peer id=f060f60df76db002 conn=trusted-staticdial peerID=f060f60df76db002 peerName=Klaytn/v1.12.1/linux-amd64/go1.20.6 err="Invalid message - msg #11 (5220 bytes): invalid message: (code b) (size 5220) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
ERROR[03/27,07:39:37 Z] [33] Protocol istanbul/65 failed id=f060f60df76db002 conn=trusted-staticdial err="Invalid message - msg #11 (5220 bytes): invalid message: (code b) (size 5220) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
ERROR[03/27,07:39:40 Z] [33] ProtocolManager failed to handle message id=e3b3d4921056ab94 conn=dyndial msg="msg #11 (6414 bytes)" err="Invalid message - msg #11 (6414 bytes): invalid message: (code b) (size 6414) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
ERROR[03/27,07:39:41 Z] [33] Protocol istanbul/65 failed id=e3b3d4921056ab94 conn=dyndial err="Invalid message - msg #11 (6414 bytes): invalid message: (code b) (size 6414) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
ERROR[03/27,07:39:47 Z] [33] ProtocolManager failed to handle message id=301c8d7d8f3b0352 conn=trusted-staticdial msg="msg #8 (2805 bytes)" err="Invalid message - msg msg #8 (2805 bytes): invalid message: (code 8) (size 2805) rlp: input list has too many elements for types.Header, decoding into ([]*types.Header)[0]"
INFO[03/27,07:39:47 Z] [33] Disconnected a multichannel P2P Peer id=301c8d7d8f3b0352 conn=trusted-staticdial peerID=301c8d7d8f3b0352 peerName=Klaytn/v1.12.1/linux-amd64/go1.20.6 err="Invalid message - msg msg #8 (2805 bytes): invalid message: (code 8) (size 2805) rlp: input list has too many elements for types.Header, decoding into ([]*types.Header)[0]"
ERROR[03/27,07:39:47 Z] [33] Protocol istanbul/65 failed id=301c8d7d8f3b0352 conn=trusted-staticdial err="Invalid message - msg msg #8 (2805 bytes): invalid message: (code 8) (size 2805) rlp: input list has too many elements for types.Header, decoding into ([]*types.Header)[0]"
usa-palchaudhary@usazuadvpl00073:/NewDisk116Machine$ tail /NewDisk116Machine/node-klay/kend/data/logs/kend.out
INFO[03/27,07:40:04 Z] [33] Added a multichannel P2P Peer id=301c8d7d8f3b0352 conn=trusted-staticdial peerID=301c8d7d8f3b0352
INFO[03/27,07:40:11 Z] [33] Added a single channel P2P Peer id=e3b3d4921056ab94 conn=dyndial peerID=e3b3d4921056ab94
ERROR[03/27,07:40:11 Z] [33] ProtocolManager failed to handle message id=f060f60df76db002 conn=trusted-staticdial msg="msg #11 (4815 bytes)" err="Invalid message - msg #11 (4815 bytes): invalid message: (code b) (size 4815) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
INFO[03/27,07:40:11 Z] [33] Disconnected a multichannel P2P Peer id=f060f60df76db002 conn=trusted-staticdial peerID=f060f60df76db002 peerName=Klaytn/v1.12.1/linux-amd64/go1.20.6 err="Invalid message - msg #11 (4815 bytes): invalid message: (code b) (size 4815) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
ERROR[03/27,07:40:11 Z] [33] Protocol istanbul/65 failed id=f060f60df76db002 conn=trusted-staticdial err="Invalid message - msg #11 (4815 bytes): invalid message: (code b) (size 4815) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
WARN[03/27,07:40:19 Z] [40] Failed doConnTypeHandshake addr=3.37.16.97:32323 conn=dyndial conntype=-1 err="read tcp 10.213.9.116:33740->3.37.16.97:32323: read: connection reset by peer"
ERROR[03/27,07:40:21 Z] [33] ProtocolManager failed to handle message id=301c8d7d8f3b0352 conn=trusted-staticdial msg="msg #11 (8120 bytes)" err="Invalid message - msg #11 (8120 bytes): invalid message: (code b) (size 8120) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
INFO[03/27,07:40:21 Z] [33] Disconnected a multichannel P2P Peer id=301c8d7d8f3b0352 conn=trusted-staticdial peerID=301c8d7d8f3b0352 peerName=Klaytn/v1.12.1/linux-amd64/go1.20.6 err="Invalid message - msg #11 (8120 bytes): invalid message: (code b) (size 8120) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
ERROR[03/27,07:40:21 Z] [33] Protocol istanbul/65 failed id=301c8d7d8f3b0352 conn=trusted-staticdial err="Invalid message - msg #11 (8120 bytes): invalid message: (code b) (size 8120) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
ERROR[03/27,07:40:26 Z] [33] ProtocolManager failed to handle message id=e3b3d4921056ab94 conn=dyndial msg="msg #11 (4627 bytes)" err="Invalid message - msg #11 (4627 bytes): invalid message: (code b) (size 4627) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"

[hyunsooda]

Hi @mayank-daga,

I guess you're running outdated node. Could you let me know which node version you're running. If it's not a latest one, please replace with the latest one. Header struct was recently updated with the hardfork.

[mayank-daga]

Hi @hyunsooda, I am using snapshot approach. I downloaded the latest snapshot and updated Data_DIR but i am still getting same error

INFO[04/03,07:37:12 Z] [33] [Dial] Add dial candidate from static nodes id=10035abc98acdd40 NodeType=2 ip=18.139.72.99 mainPort=32323 port="[32323 32324]"
INFO[04/03,07:37:13 Z] [33] Added a multichannel P2P Peer id=10035abc98acdd40 conn=trusted-staticdial peerID=10035abc98acdd40
INFO[04/03,07:37:14 Z] [33] [Dial] Add dial candidate from static nodes id=57d667c3783c3bc8 NodeType=2 ip=3.1.171.124 mainPort=32323 port="[32323 32324]"
INFO[04/03,07:37:15 Z] [33] Added a multichannel P2P Peer id=57d667c3783c3bc8 conn=trusted-staticdial peerID=57d667c3783c3bc8
ERROR[04/03,07:37:15 Z] [33] ProtocolManager failed to handle message id=57d667c3783c3bc8 conn=trusted-staticdial msg="msg #11 (10984 bytes)" err="Invalid message - msg #11 (10984 bytes): invalid message: (code b) (size 10984) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
INFO[04/03,07:37:15 Z] [33] Disconnected a multichannel P2P Peer id=57d667c3783c3bc8 conn=trusted-staticdial peerID=57d667c3783c3bc8 peerName=Klaytn/v1.12.1/linux-amd64/go1.20.6 err="Invalid message - msg #11 (10984 bytes): invalid message: (code b) (size 10984) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
ERROR[04/03,07:37:15 Z] [33] Protocol istanbul/65 failed id=57d667c3783c3bc8 conn=trusted-staticdial err="Invalid message - msg #11 (10984 bytes): invalid message: (code b) (size 10984) rlp: input list has too many elements for types.Header, decoding into (cn.newBlockData).Block(types.extblock).Header"
ERROR[04/03,07:37:17 Z] [33] ProtocolManager failed to handle message id=10035abc98acdd40 conn=trusted-staticdial msg="msg #8 (2805 bytes)" err="Invalid message - msg msg #8 (2805 bytes): invalid message: (code 8) (size 2805) rlp: input list has too many elements for types.Header, decoding into ([]*types.Header)[0]"
INFO[04/03,07:37:17 Z] [33] Disconnected a multichannel P2P Peer id=10035abc98acdd40 conn=trusted-staticdial peerID=10035abc98acdd40 peerName=Klaytn/v1.12.1/linux-amd64/go1.20.6 err="Invalid message - msg msg #8 (2805 bytes): invalid message: (code 8) (size 2805) rlp: input list has too many elements for types.Header, decoding into ([]*types.Header)[0]"
ERROR[04/03,07:37:17 Z] [33] Protocol istanbul/65 failed id=10035abc98acdd40 conn=trusted-staticdial err="Invalid message - msg msg #8 (2805 bytes): invalid message: (code 8) (size 2805) rlp: input list has too many elements for types.Header, decoding into ([]*types.Header)[0]"

[hyunsooda]

No, I meant you should identify your node version. I guess you're running an outdated node version. If the node version is not the latest one, please replace it with the latest one.

[mayank-daga]

I am using v12.1.0 @hyunsooda i.e. latest only

[exalate-issue-sync]

Exalate commented: Issue Created by: mayank-daga

[mayank-daga]

Hi @hyunsooda just following up

[mayank-daga]

Hi @hyunsooda just following up any solution we have

[yoomee1313]

can you share the outputs when you checked the version as lake guided? For sure, please check both the attach welcome log and binary version.
(1) attach welcome log

$ ./ken attach klay.ipc
Welcome to the Klaytn JavaScript console!

 instance: Klaytn/v1.12.1/darwin-arm64/go1.22.1
...

(2) binary version

$ ./ken version
Klaytn v1.12.1+a88e375fea

[mayank-daga]

Hi @yoomee1313 I checked the binary version using the above command:
Klaytn v1.12.1+4831f49bba

its different where can i get latest one ?

[yoomee1313]

@mayank-daga 4831f49bba commit hash is correct. Mine was different since it's from my working directory. Btw, can I know why you replace the chain data?

Could you share the link where you downloaded and the current block number? Also please check that you correctly locate the new chaindata and the ken process is properly restarted.

On the other hands, you can check the connected peer info by admin.peers. When the node is restarted, new peers should be connected to your node automatically. However, sometimes it is not done by properly when the node chaindata is replaced with new chaindata. In many cases, nodekey was problem, but not 100%.

[mayank-daga]

Hi @yoomee1313 I downloaded the latest ken and chain data from klaytn docs. (mentioned below url)
Chaindata: http://packages.klaytn.net/cypress/chaindata/
Ken package: https://packages.klaytn.net/klaytn/v1.11.1/ken-v1.11.1-0-linux-amd64.tar.gz
I have attached kend.conf details as i couldnt find admin.peers:

Configuration file for the kend

cypress, baobab is only available if you don't specify NETWORK_ID.

NETWORK="cypress"

if you specify NETWORK_ID, a private network is created.

NETWORK_ID=

PORT=32323

SERVER_TYPE="http"
SYNCMODE="full"
VERBOSITY=3
MAXCONNECTIONS=10

txpool options setting

TXPOOL_EXEC_SLOTS_ALL=4096
TXPOOL_NONEXEC_SLOTS_ALL=4096
TXPOOL_EXEC_SLOTS_ACCOUNT=4096
TXPOOL_NONEXEC_SLOTS_ACCOUNT=4096
TXPOOL_LIFE_TIME="30m"

rpc options setting

RPC_ENABLE=1 # if this is set, the following options will be used
RPC_API="klay,eth,net,web3" # available apis: admin,debug,klay,eth,miner,net,personal,rpc,txpool,web3
RPC_PORT=8551
RPC_ADDR="0.0.0.0"
RPC_CORSDOMAIN=""
RPC_VHOSTS=""

below options are related with http server

RPC_CONCURRENCYLIMIT=3000
RPC_READ_TIMEOUT=30
RPC_WRITE_TIMEOUT=30
RPC_IDLE_TIMEOUT=120
RPC_EXECUTION_TIMEOUT=30

ws options setting

WS_ENABLE=1 # if this is set, the following options will be used
WS_API="klay,eth,net,web3" # available apis: admin,debug,klay,eth,miner,net,personal,rpc,txpool,web3
WS_ADDR="0.0.0.0"
WS_PORT=8552
WS_ORIGINS="*"

service chain options setting

SC_MAIN_BRIDGE=0 # if this is set, the following options will be used.
SC_MAIN_BRIDGE_PORT=50505
SC_MAIN_BRIDGE_INDEXING=0 # this option will be deprecated.

Setting 1 is to enable options, otherwise disabled.

AUTO_RESTART=0
METRICS=1
PROMETHEUS=1
DB_NO_PARALLEL_WRITE=0
MULTICHANNEL=1
SUBPORT=$((PORT + 1)) # used for multi channel option

discover options

NO_DISCOVER=0 # setting 1 to disable discovery
BOOTNODES=""

log rotation related options

LOG_ROTATE=0 # setting 1 to enable the log rotation related options
LOG_MAXSIZE=100 # the unit is MB
LOG_MAXBACKUPS=10
LOG_MAXAGE=30 # maximum number of days to retain a log file
LOG_COMPRESS=1 # setting 1 to compress the backup logs in gz format

Raw options e.g) "--txpool.nolocals"

ADDITIONAL=""

auto restart options

AUTO_RESTART_NODE=0
AUTO_RESTART_INTERVAL=0.1

DATA_DIR=/node-klay/v12.1.0/kend/data
LOG_DIR=$DATA_DIR/logs

[yoomee1313]

https://packages.klaytn.net/klaytn/v1.11.1/ken-v1.11.1-0-linux-amd64.tar.gz link downloads the v1.11.1 version of ken. If you're using v1.12.1 version of ken, you should download it from https://packages.klaytn.net/klaytn/v1.12.1/ken-v1.12.1-0-linux-amd64.tar.gz unless you build your own binary in local. And data directory is /node-klay/v12.1.0/kend/data. Is v12.1.0 typo?

Currently, admin hasn't enabled in rpc so you have to attach by IPC. klay.ipc is located under data directory.

$ ./ken attach klay.ipc