Inconsistent debug_storageRangeAt due to missing hash to preimage map in state migrated db

[swapscanner-ryan]

Describe the bug
The debug_storageRangeAt JSON-RPC method is expected to return the key-value pairs of the entire contract storage. However, an inconsistency in the method's response has been observed, where the key is not correctly resolved from a state trie migrated database. This results in some entries having an empty string as a key. This behavior was noted by comparing responses from two different nodes:

• One node had its state trie migrated database recently downloaded from https://packages.klaytn.net/cypress/chaindata/
• Another node downloaded its state trie migrated database about a year ago and did not undergo any migration thereafter.

How to reproduce
The issue can be reproduced using the following curl command:

$ curl --compressed -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"debug_storageRangeAt","params":["CURRENT_BLOCK_HASH",0,"0x10dd79cf8bea084cfee1099a30fc3be6a88df2ba","0x0000000000000000000000000000000000000000000000000000000000000000",10000000000],"id":1}' \
  $NODE:8551

Further investigation reveals that the secure trie hash to preimage map is missing in the state trie migrated database, as demonstrated by the following proof:

$ curl -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"debug_preimage","params":["0xf934003e0f1e08fb7a6085c2e0ac1cad9804650495759671529d73156a5a6cc2"],"id":"1"}' $RECENTLY_STATEMIGRATED_NODE:8551
{"jsonrpc":"2.0","id":"1","error":{"code":-32000,"message":"unknown preimage"}}

$ curl -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"debug_preimage","params":["0xf934003e0f1e08fb7a6085c2e0ac1cad9804650495759671529d73156a5a6cc2"],"id":"1"}' $NEVER_STATEMIGRATED_NODE:8551
{"jsonrpc":"2.0","id":"1","result":"0x0a73c8aefe69b00ca092343aab2f3b9a36211f58dff249d7becceba06c8cb3f7"}

Expected behavior
The debug_storageRangeAt method should correctly return all key-value pairs of the contract storage, including resolving keys correctly from the state trie migrated database.

Attachments
N/A

Environment (please complete the following information)

• Klaytn version or git revision that exhibits the issue $ ken version: 1.12.1
• go version $ go version: 1.21.6 (but not Go version dependent)
• OS and its version [e.g. macOS/10.14]: CentOS 7 (but not os dependent)

Additional context
The absence of the hash -> preimage map in the state trie migrated database seems to prevent the secure trie from resolving the actual keys for the stored values. This issue does not affect typical operations (e.g., SLOAD, SSTORE), where preimage -> hash is used, but impedes the correct functioning of the debug_storageRangeAt method. My theory is that the mapping cache of statedb.SecureTrie that stores hash -> preimage mappings is not copied during state trie migration, indicating a specific issue with the migration process and the handling of secure tries.

[exalate-issue-sync]

Exalate commented: Issue Created by: swapscanner-ryan

[blukat29]

Hello @swapscanner-ryan, thank you for the report.

The debug_storageRangeAt may not return the preimages of an entry in the secure Merkle Patricia Tree, in which case the key field of the storage item is null.

• When the preimage exists in DB

   {
     nextKey: null,
     storage: {
       0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563: {  // secure key (hash)
         key: "0x0000000000000000000000000000000000000000000000000000000000000000",  // key (preimage)
         value: "0x000000000000000000000000000000000000000000000000000000000000007b"
       },

• When the preimage does not exist in DB

   {
     nextKey: null,
     storage: {
       0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e563: { // secure key (hash)
         key: null,      // key (preimage) not found
         value: "0x000000000000000000000000000000000000000000000000000000000000007b"
       },

As you have discovered, state-migrated databases do not contain secure MPT preimages. The state migration process currently does not copy the preimages. Therefore the nodes started from a state-migrated database will have partial preimage data.

We are evaluating the preservation of preimages during state migration (#2112) regarding storage costs and preimage uses.

For now, please use a database that is not state-migrated to access the complete preimage database.