Missing trie node after initial snap sync

[rjl493456442]

The original report is from #29417

Geth:

• scheme: hash
• version: 1.13.14-stable

reports error like this:

Apr 01 18:04:59 aviad-SEi geth[69608]: ERROR[04-01|18:04:59.141] Unhandled trie error in Trie.Prove       err="missing trie node 67120b27ad8a3045df012b9e4e2daae8ed047f5e52bc7800356b6b9cd633709c (path 0000010a060a0b) <nil>"

The error means an account trie node at position []byte{0x0, 0x0, 0x1, 0xa, 0x6, 0xa, 0xb} is missing.

The correct node blob should be 0xf8669d35fce6efd2272cdb9ea2eacce639fab06c112ac8be435a182bae332defb846f8440180a0bbd13d463ce2111713bb0d6ad117b0ba991bc32336d501e872703c4ad6bc3c90a0905982d3bc98b3ad75d5ca7a92446cc8923fae39a2352245000f9d3e9dc0f3ee, can be decoded as a shortNode:

[
  35fce6efd2272cdb9ea2eacce639fab06c112ac8be435a182bae332def,
  f8440180a0bbd13d463ce2111713bb0d6ad117b0ba991bc32336d501e872703c4ad6bc3c90a0905982d3bc98b3ad75d5ca7a92446cc8923fae39a2352245000f9d3e9dc0f3ee,
]

After inspecting the history (from block 19400000, roughly Mar-10-2024), the account mutation with address-hash prefix 001a6a is found as below:

./geth db inspect-history --datadir . --start 19400000 --prefix 001a6a --datadir ./geth_data

Account history:

        blockrange: [#19400000-#19600717]
#19541282
                001a6aefc178724fdc1d7f0f7b655fca3dfb9d5ae36b2425d9efe98c1d0e7126 - (address: 3b675ffd72a3be289d1cb3e8cceb0417505d5fcd): 0xcd820121871ad7eef6b7da568080

It's a sibling of missing node mutated on Mar-29-2024

[Luckymhe]

How to work on it

[rjl493456442]

Another similar report is from Geth-discord server:

Geth:

scheme: hash
version: unknown

Mar 19 11:25:48 OrangePi5P geth[15795]: INFO [03-19|11:25:48.482] Resuming state snapshot generation       root=f3b387..c43e63 in=1df820..0b7a24 at=290dec..f3e563 accounts=31,432,500
slots=202,708,968             storage=15.88GiB   dangling=0 elapsed=4h45m37.902s eta=35h54m15.615s
Mar 19 11:25:48 OrangePi5P geth[15795]: INFO [03-19|11:25:48.549] Imported new potential chain segment     number=19,468,484 hash=90d01d..445b7a blocks=1         txs=174         mgas=12.808  elapsed=241.250ms    mgasps=53.091  snapdiffs=354.57KiB triedirty=802.61MiB
Mar 19 11:25:48 OrangePi5P geth[15795]: INFO [03-19|11:25:48.641] Chain head was updated                   number=19,468,484 hash=90d01d..445b7a root=73dd6f..e436c2 elapsed=11.101605ms
Mar 19 11:25:48 OrangePi5P geth[15795]: ERROR[03-19|11:25:48.804] Unhandled trie error in Trie.Prove       err="missing trie node e6fdf16ed01e73368d8ee4f80b9f6d3f54fe7eb335397503de6d0bb3bc5e4ec1 (path 010d0f0a060e) <nil>"
Mar 19 11:25:48 OrangePi5P geth[15795]: ERROR[03-19|11:25:48.811] State snapshotter failed to iterate trie err="missing trie node e6fdf16ed01e73368d8ee4f80b9f6d3f54fe7eb335397503de6d0bb3bc5e4ec1 (path 010d0f0a060e) <nil>"

The missing node is a fullNode in account trie

0xf90111a08e0ba3b973c207759cc214d0cdab06c9955b580dcc1eb6e68590b87a277fb09c808080a0f252843b56a5a339c7413cd233b82ff8bf034e6f8bfda7e77558671e318dc8afa0c598146e383e0aebf94fa73900bfeb9a0731d1fb52c58c365ca0528f08160f20808080a093494ce0ff9969d9c3ba3d965118557bd376a58877aeac1918ba466d98c009b4a05ad20e32eb2c575bbd6ad27094223739d82515132c7ec92b69fc8f1d44da62b9a01d7e731a5d22d8db4e107aeb749bf25e985d1a57b5e658e287d7e279056b5b8da0f0b1e09ee6d8af2ae08fd8cf698ff982e2387408d637d3f468d132de0dec86c080a0a6f35629bb9ef5fe1c36354098af4c4cbef0975779b8da89b3bea9013f899f8f8080

[
  8e0ba3b973c207759cc214d0cdab06c9955b580dcc1eb6e68590b87a277fb09c,
  "",
  "",
  "",
  f252843b56a5a339c7413cd233b82ff8bf034e6f8bfda7e77558671e318dc8af,
  c598146e383e0aebf94fa73900bfeb9a0731d1fb52c58c365ca0528f08160f20,
  "",
  "",
  "",
  93494ce0ff9969d9c3ba3d965118557bd376a58877aeac1918ba466d98c009b4,
  5ad20e32eb2c575bbd6ad27094223739d82515132c7ec92b69fc8f1d44da62b9,
  1d7e731a5d22d8db4e107aeb749bf25e985d1a57b5e658e287d7e279056b5b8d,
  f0b1e09ee6d8af2ae08fd8cf698ff982e2387408d637d3f468d132de0dec86c0,
  "",
  a6f35629bb9ef5fe1c36354098af4c4cbef0975779b8da89b3bea9013f899f8f,
  "",
  "",
]

After inspecting the history (from block 19400000, roughly Mar-10-2024), the account mutation with address-hash prefix 1dfa6 is found as below.

Account history:

        blockrange: [#19400000-#19600717]
#19408862
                1dfa67ab98fa17e1e07333e8775afd82eb67507342d65166e4ea61b43f22101f - (address: eedf737b1ae162bdc6c3c1f9d9710641cbdf9c9c): 0xcc8088074f490ca01800008080
#19455625
                1dfa69bc89f51034e41e3e58be59d18bd13e9ff327331b0017e88c97cdddc3f8 - (address: 0b583b9a448e44bf7c6975d207f8f3e4d43778f4):
#19484010
                1dfa633edfe0436f98846530f6e939f915628ab4854a5854238fade9c4bddc59 - (address: 440bdd191dbdf7cadef6fd9d901f24782dab77ae):
#19484034
                1dfa633edfe0436f98846530f6e939f915628ab4854a5854238fade9c4bddc59 - (address: 440bdd191dbdf7cadef6fd9d901f24782dab77ae): 0xcb8087c50554634433948080
#19484163
                1dfa63df42783eef34d7e4b15b128bc62f3452a91ea3b58e93cad290bac2e4cf - (address: bbb82009b3af274c7b7f392f3dfa326c60950475): 0xcc22880137c9f8941b7ab78080
#19491753
                1dfa60e6344cd434a8df6e181cd25e9cc84e567e7d554085a6598196bf32f00d - (address: cc6952a0108d0d5504457a7aa5383244c0046bd3): 0xcb01870876569eb15c6b8080
#19512048
                1dfa633edfe0436f98846530f6e939f915628ab4854a5854238fade9c4bddc59 - (address: 440bdd191dbdf7cadef6fd9d901f24782dab77ae): 0xcb0187012275f4d82a608080
#19519217
                1dfa633edfe0436f98846530f6e939f915628ab4854a5854238fade9c4bddc59 - (address: 440bdd191dbdf7cadef6fd9d901f24782dab77ae): 0xcc018802c7ad66afec2a608080
#19569204
                1dfa620c24104c229bb057896297422a1b27a8d4167ef6a8cd70dbff397d3ab6 - (address: a620260d42315a2532852339f4b7ae4db5e40407): 0xcb0887879f2b906f1f5e8080
#19569219
                1dfa620c24104c229bb057896297422a1b27a8d4167ef6a8cd70dbff397d3ab6 - (address: a620260d42315a2532852339f4b7ae4db5e40407): 0xcb0887e395f0d4ffeb5e8080
#19569305
                1dfa620c24104c229bb057896297422a1b27a8d4167ef6a8cd70dbff397d3ab6 - (address: a620260d42315a2532852339f4b7ae4db5e40407): 0xcc0888b54cdd105513eb5e8080
#19580498
                1dfa6191a48f0a3c1ac6ad696c0f7a8bd16573865316d709e499452346a75c98 - (address: e927064b56920f9edf3930d3f23fdfabc18d8290):
#19587242
                1dfa6385c300e5310e3baec66ed016e0f97c4a765e1aa3a8350f8078de2b081e - (address: 87ba13643bca4c44c110066150242b216ecaf53d): 0xcc808803358f939ad408008080
#19587270
                1dfa6385c300e5310e3baec66ed016e0f97c4a765e1aa3a8350f8078de2b081e - (address: 87ba13643bca4c44c110066150242b216ecaf53d): 0xcc0188032a89cdb065fc348080

Several siblings are mutated, especially the one in block 19455625 which is on Mar-17-2024