Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core/types: remove support for legacy receipt/log storage encoding #22852

Merged
merged 2 commits into from May 11, 2021

Conversation

fjl
Copy link
Contributor

@fjl fjl commented May 10, 2021

This removes the database upgrade path for receipts and logs stored before April 2019.

@fjl fjl added this to the 1.10.4 milestone May 10, 2021
@fjl fjl changed the title core/types: remove support for legacy storage encodings core/types: remove support for legacy receipt/log storage encodings May 10, 2021
@fjl fjl changed the title core/types: remove support for legacy receipt/log storage encodings core/types: remove support for legacy receipt/log storage encoding May 10, 2021
Copy link
Contributor

@holiman holiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but let's hear what karalabe thinks too

Copy link
Member

@karalabe karalabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Safe to delete as we introduced the freezer at the same time as the new format and the freezer moved everything reencoding, so nobody should have any old format stuff any more.

@karalabe karalabe merged commit 643fd0e into ethereum:master May 11, 2021
@shoenseiwaso
Copy link
Contributor

shoenseiwaso commented Jun 24, 2021

Note that this comment is intended only to help guide others who have the same problem, we are not expecting a fix.

For anyone who sees the following in their geth console logs with v1.10.4 or later:

ERROR[06-22|15:02:49.899] Invalid receipt array RLP                hash=28c41d..6bb895 err="rlp: expected input list for []*types.LogForStorage, decoding into ([]*types.ReceiptForStorage)[0](types.storedReceiptRLP).Logs"
ERROR[06-22|15:02:49.899] Invalid receipt array RLP                hash=506213..15eea1 err="rlp: expected input list for []*types.LogForStorage, decoding into ([]*types.ReceiptForStorage)[0](types.storedReceiptRLP).Logs"
ERROR[06-22|15:02:49.899] Invalid receipt array RLP                hash=fb8a03..3e976e err="rlp: expected input list for []*types.LogForStorage, decoding into ([]*types.ReceiptForStorage)[0](types.storedReceiptRLP).Logs"
ERROR[06-22|15:02:49.901] Invalid receipt array RLP                hash=506213..15eea1 err="rlp: expected input list for []*types.LogForStorage, decoding into ([]*types.ReceiptForStorage)[0](types.storedReceiptRLP).Logs"

Or sees the following error when they attempt to export their chaindata:

retrieved hash chain is invalid: invalid gas used (remote: 1884897 local: 1542885)

What probably happened is that at some point in the past, your geth database (chaindata) got silently corrupted and failed to automatically upgrade to the modern receipt/log storage encoding. With this change (#22852) go-ethereum removes the capability to read these legacy logs.

Possible solutions include:
(1) Recreate your geth database (chaindata) from scratch
(2) Postpone the upgrade to v1.10.4 or build a custom geth without this PR and run it until you can do (1)

fjl added a commit that referenced this pull request Jul 7, 2021
The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like #17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in #22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>
sidhujag pushed a commit to sidhujag/go-ethereum that referenced this pull request Jul 10, 2021
The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like ethereum#17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in ethereum#22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>
yehjxraymond added a commit to yehjxraymond/go-ethereum that referenced this pull request Jul 13, 2021
* eth/tracers: expose contextual infos (block hash, tx hash, tx index)

* eth/tracers: convert int/hash values from context into js object (ethereum#23108)

* Convert int/hash values from context into js object

* Use js fixed buffer

Co-authored-by: William <william.berman@coinbase.com>

* cmd/geth: fix IPC probe in les test (ethereum#23094)

Previously, the test waited a second and then failed if geth had not
started. This caused the test to fail intermittently. This change checks
whether the IPC is open 10 times over a 5 second period and then fails
if geth is still not available.

* eth/gasprice: fix typo in comment (ethereum#22998)

* core/state: remove unused methods ReturnGas, GetStorageProofByHash (ethereum#23092)

Co-authored-by: lidongwei <lidongwei@huobi.com>

* eth/downloader: increase downloader block body allowance (ethereum#23074)

This change increases the cache size from 64 to 256 Mb for block bodies.
Benchmarks have shown this to be one bottleneck when trying to achieve
higher download speeds.

The commit also includes a minor optimization for header inserts in package
core: previously, the presence of headers in the database was checked for
every header before writing it. With the change, if one header fails the
presence check, all subsequent headers are also assumed to be missing.
This is an improvement because in practice, the headers are almost always
missing during sync.

* p2p/enode: fix method doc (ethereum#23115)

This is an obvious spelling error

Co-authored-by: liuyaxiong <liuyaxiong@inspur.com>

* core, eth: fix precompile addresses for tracers (ethereum#23097)

* core,eth/tracers: make isPrecompiled dependent on HF

* eth/tracers: use keys when constructing chain config struct

* eth/tracers: dont initialize activePrecompiles with random value

* eth/gasprice: implement feeHistory API (ethereum#23033)

* eth/gasprice: implement feeHistory API

* eth/gasprice: factored out resolveBlockRange

* eth/gasprice: add sanity check for missing block

* eth/gasprice: fetch actual gas used from receipts

* miner, eth/gasprice: add PendingBlockAndReceipts

* internal/ethapi: use hexutil.Big

* eth/gasprice: return error when requesting beyond head block

* eth/gasprice: fixed tests and return errors correctly

* eth/gasprice: rename receiver name

* eth/gasprice: return directly if blockCount == 0

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>

* accounts/abi/bind: fix gas price suggestion with pre EIP-1559 clients (ethereum#23102)

This fixes transaction sending in the case where an app using go-ethereum v1.10.4
is talking to a pre-EIP-1559 RPC node. In this case, the eth_maxPriorityFeePerGas
endpoint is not available and we can only rely on eth_gasPrice.

* p2p/server: fix method name in comment (ethereum#23123)

* ethclient/gethclient: RPC client wrapper for geth-specific API (ethereum#22977)

This commit adds the package gethclient which is similar to the ethclient
and implements some geth specific functionality.

Co-authored-by: Edgar Aroutiounian <edgar.factorial@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>

* cmd/geth: dont fail on deprecated toml config fields (ethereum#23118)

* internal/ethapi: fix panic in access list creation (ethereum#23133)

Fixes test failure in the last commit.

* all: removed blockhash from statedb (ethereum#23126)

This PR removes the blockhash from the statedb

* eth/fetcher, trie: unit test reliability fixes (ethereum#23020)

Some tests take quite some time during exit, which I think causes
some appveyor fails like this:

    https://ci.appveyor.com/project/ethereum/go-ethereum/builds/39511210/job/xhom84eg2e4uulq3

One of the things that seem to take time during exit is waiting
(up to 100ms) for the syncbloom to close. This PR changes it to use
a channel, instead of looping with a 100ms wait.

This also includes some unrelated changes improving the reliability of
eth/fetcher tests, which fail a lot because they are time-dependent.

* crypto: gofuzz build directives (ethereum#23137)

* eth/tracers: improve tracing performance (ethereum#23016)

Improves the performance of debug.traceTransaction

* go.mod: update UPNP dependency (ethereum#23116)

* les: avoid shutdown hang (ethereum#23139)

* consensus/clique: avoid a copy in clique (ethereum#23149)

* consensus/clique:optimize to avoid a copy in clique

* consensus/clique: test for sealhash

Co-authored-by: Martin Holst Swende <martin@swende.se>

* cmd/evm, eth/ethconfig: regenerate struct codecs (ethereum#23140)

* ethstats: fix full node interface post 1559

* dockerfile: get rid of make and env, see if that fixes builds

* cmd/puppeth: fix dashboard crash caused by updated base image

* core: fix bad parent hash when jumping to genesis in setHead (ethereum#23162)

* p2p: fix array out of bounds issue (ethereum#23165)

* fuzzing: fix typo in fuzzer definitions (ethereum#23169)

* eth, miner: add RPC method to modify miner gaslimit (pre london: ceiling) (ethereum#23134)

* core/types: sanity check the basefee length inside a header (ethereum#23171)

* all: remove noop vm config flags (ethereum#23111)

* all: rm external interpreter and ewasm config

* core/vm: rm Interpreter interface

* cmd/geth: deprecate interpreter config fields

* cmd/devp2p: fixes for eth and discv4 tests (ethereum#23155)

This PR fixes a false positive PONG 'to' endpoint mismatch seen in hive tests:

    got {IP:172.17.0.7 UDP:44025 TCP:44025}, want {IP:172.17.0.7 UDP:44025 TCP:0}

Co-authored-by: Felix Lange <fjl@twurst.com>

* core/types: remove LogForStorage type (ethereum#23173)

The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like ethereum#17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in ethereum#22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>

* core/types: go generate (ethereum#23177)

* core, params: define london block at 12965000 (ethereum#23176)

* core, params: define london block at 12965000

* core/forkid: fix test

* eth/gasprice, internal/ethapi, miner: minor feehistory fixes

* params: update CHTs for the 1.10.5 release

* core/state: avoid unnecessary alloc in trie prefetcher (ethereum#23198)

* internal/ethapi: fix transaction APIs (ethereum#23179)

* internal/ethapi: fix transaction APIs

* internal/ethapi: fix typo

* internal/ethapi: address comments

* internal/ethapi: address comment from Peter

* internal: get pending and queued transaction by address (ethereum#22992)

* core, eth, internal, les, light: get pending and queued transaction by address

* core: tiny nitpick fixes

* light: tiny nitpick

Co-authored-by: mark <mark@amis.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>

* consensus/clique: implement getSigner API method (ethereum#22987)

* clique: implement getSignerForBlock

* consensus/clique: use blockNrOrHash in getSignerForBlock

* consensus/clique: implement getSigner

* consensus/clique: fixed rlp decoding

* consensus/clique: use Author instead of getSigner

* consensus/clique: nit nit nit

* consensus/clique: nit nit nit

* feat: merged changes from head

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: williamberman <wlbberman@gmail.com>
Co-authored-by: William <william.berman@coinbase.com>
Co-authored-by: piersy <pierspowlesland@gmail.com>
Co-authored-by: haryu703 <34744512+haryu703@users.noreply.github.com>
Co-authored-by: Li Dongwei <lidw1988@126.com>
Co-authored-by: lidongwei <lidongwei@huobi.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: ForLina <471133417@qq.com>
Co-authored-by: liuyaxiong <liuyaxiong@inspur.com>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Felföldi Zsolt <zsfelfoldi@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Pierre R <p.rousset@gmail.com>
Co-authored-by: Ahyun <urbanart2251@gmail.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Edgar Aroutiounian <edgar.factorial@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: ucwong <ucwong@126.com>
Co-authored-by: Evolution404 <35091674+Evolution404@users.noreply.github.com>
Co-authored-by: Mark <markya0616@gmail.com>
Co-authored-by: mark <mark@amis.com>
Co-authored-by: Raymond Yeh <ray@geek.sg>
@rjl493456442
Copy link
Member

@shoenseiwaso not sure you are still around. Are you willing to recompile the geth with these diffs? I am curious why it's failed. Since we have the freezer to re-encode the legacy receipts. No legacy format receipts should be left.

diff --git a/core/rawdb/accessors_chain.go b/core/rawdb/accessors_chain.go
index 76132bf37..871ab0b64 100644
--- a/core/rawdb/accessors_chain.go
+++ b/core/rawdb/accessors_chain.go
@@ -572,7 +572,7 @@ func ReadRawReceipts(db ethdb.Reader, hash common.Hash, number uint64) types.Rec
        // Convert the receipts from their storage form to their internal representation
        storageReceipts := []*types.ReceiptForStorage{}
        if err := rlp.DecodeBytes(data, &storageReceipts); err != nil {
-               log.Error("Invalid receipt array RLP", "hash", hash, "err", err)
+               log.Error("Invalid receipt array RLP", "number", number, "hash", hash.Hex(), "err", err)
                return nil
        }
        receipts := make(types.Receipts, len(storageReceipts))

@rjl493456442
Copy link
Member

Let's discuss at #23245

rjl493456442 added a commit to rjl493456442/go-ethereum that referenced this pull request Jul 22, 2021
fjl pushed a commit that referenced this pull request Jul 22, 2021
* Revert "core/types: go generate (#23177)"

This reverts commit 00b922f.

* Revert "core/types: remove LogForStorage type (#23173)"

This reverts commit 7522642.

* Revert "core/types: remove support for legacy receipt/log storage encoding (#22852)"

This reverts commit 643fd0e.
jwasinger pushed a commit to jwasinger/go-ethereum that referenced this pull request Jul 25, 2021
…thereum#22852)

* core/types: remove support for legacy receipt storage encoding

* core/types: remove support for legacy log storage encoding
sidhujag pushed a commit to sidhujag/go-ethereum that referenced this pull request Jul 28, 2021
* Revert "core/types: go generate (ethereum#23177)"

This reverts commit 00b922f.

* Revert "core/types: remove LogForStorage type (ethereum#23173)"

This reverts commit 7522642.

* Revert "core/types: remove support for legacy receipt/log storage encoding (ethereum#22852)"

This reverts commit 643fd0e.
reds pushed a commit to reds/go-ethereum that referenced this pull request Aug 28, 2021
…thereum#22852)

* core/types: remove support for legacy receipt storage encoding

* core/types: remove support for legacy log storage encoding
reds pushed a commit to reds/go-ethereum that referenced this pull request Aug 28, 2021
The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like ethereum#17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in ethereum#22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>
reds pushed a commit to reds/go-ethereum that referenced this pull request Aug 28, 2021
* Revert "core/types: go generate (ethereum#23177)"

This reverts commit 00b922f.

* Revert "core/types: remove LogForStorage type (ethereum#23173)"

This reverts commit 7522642.

* Revert "core/types: remove support for legacy receipt/log storage encoding (ethereum#22852)"

This reverts commit 643fd0e.
i-norden pushed a commit to cerc-io/go-ethereum that referenced this pull request Sep 10, 2021
The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like ethereum#17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in ethereum#22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>
i-norden pushed a commit to cerc-io/go-ethereum that referenced this pull request Sep 10, 2021
* Revert "core/types: go generate (ethereum#23177)"

This reverts commit 00b922f.

* Revert "core/types: remove LogForStorage type (ethereum#23173)"

This reverts commit 7522642.

* Revert "core/types: remove support for legacy receipt/log storage encoding (ethereum#22852)"

This reverts commit 643fd0e.
atif-konasl pushed a commit to frozeman/pandora-execution-engine that referenced this pull request Oct 15, 2021
…thereum#22852)

* core/types: remove support for legacy receipt storage encoding

* core/types: remove support for legacy log storage encoding
atif-konasl pushed a commit to frozeman/pandora-execution-engine that referenced this pull request Oct 15, 2021
The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like ethereum#17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in ethereum#22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>
atif-konasl pushed a commit to frozeman/pandora-execution-engine that referenced this pull request Oct 15, 2021
* Revert "core/types: go generate (ethereum#23177)"

This reverts commit 00b922f.

* Revert "core/types: remove LogForStorage type (ethereum#23173)"

This reverts commit 7522642.

* Revert "core/types: remove support for legacy receipt/log storage encoding (ethereum#22852)"

This reverts commit 643fd0e.
holiman pushed a commit that referenced this pull request Dec 3, 2022
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt #22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted #23247. Since then we added a command to migrate legacy dbs #24028.

As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
shekhirin pushed a commit to shekhirin/go-ethereum that referenced this pull request Jun 6, 2023
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt ethereum#22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted ethereum#23247. Since then we added a command to migrate legacy dbs ethereum#24028.

As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
MoonShiesty pushed a commit to MoonShiesty/go-ethereum that referenced this pull request Aug 30, 2023
This PR drops the legacy receipt types, the freezer-migrate command and the startup check. The previous attempt ethereum#22852 at this failed because there were users who still had legacy receipts in their db, so it had to be reverted ethereum#23247. Since then we added a command to migrate legacy dbs ethereum#24028.

As of the last hardforks all users either must have done the migration, or used the --ignore-legacy-receipts flag which will stop working now.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants