Legacy receipt converter tool #23454

s1na · 2021-08-24T15:18:04Z

See #23245 and #23247 for context. Adds a command db freezer-migrate that goes through the list of receipts in the freezer and converts the legacy ones.

holiman · 2021-08-24T15:33:45Z

A thought: the first few thousand mainnet blocks were empty, so no use just checking block 1 for format.

s1na · 2021-08-24T15:36:54Z

Ah right. I should check blocks from num 1 until a non-empty receipt is found then check if its legacy

holiman · 2021-08-24T15:59:44Z

That's it, yep

MariusVanDerWijden · 2021-09-06T15:32:17Z

Do you have some numbers on how much space this saves on mainnet?

I'll sync my node up and can do a receipt conversion

s1na · 2021-11-18T16:25:00Z

To test this I did a full sync on an old commit for ~4mil blocks, then switched to version v1.9.0 for a while. Output of the migrate command:

INFO [11-18|13:07:37.018] First legacy receipt                     number=46147
INFO [11-18|13:07:37.018] Starting migration                       ancients=4,207,670
INFO [11-18|13:07:37.026] Processing legacy elements               number=0
INFO [11-18|13:07:43.214] Processing legacy elements               number=500,000
INFO [11-18|13:08:04.514] Processing legacy elements               number=1,000,000
INFO [11-18|13:08:41.157] Processing legacy elements               number=1,500,000
INFO [11-18|13:09:18.377] Processing legacy elements               number=2,000,000
INFO [11-18|13:09:57.718] Processing legacy elements               number=2,500,000
INFO [11-18|13:10:38.356] Processing legacy elements               number=3,000,000
INFO [11-18|13:11:48.204] Processing legacy elements               number=3,500,000
INFO [11-18|13:14:09.616] Found first non-legacy element           number=3,931,786 filenum=2 copyOver=true
INFO [11-18|13:14:21.536] Replacing table files
INFO [11-18|13:14:21.869] Migration finished                       duration=6m44.851466067s

Now to make sure the content of the files are correct I'm concatenating the table files and comparing that against the freezer of a node with a "modern" db. i.e.:

$ cmp -b ~/datadir-mainnet/geth/chaindata/ancient/receipts.part.cdat ~/datadir-receipts-backup/geth/chaindata/ancient/receipts.all.cdat
differ: byte 3312800212, line 12334563 is 257 M-/ 276 M->

But when I retrieve the encoded receipt for the corresponding block they seem to match. Wondering if it's snappy-related.

s1na · 2021-11-19T11:32:26Z

So I wrote a small script that compares the tables of a certain kind against that of another freezer and the result seems to match. My hunch as to the differences in the binary files is due to different snappy output. From this thread:

... Is snappy deterministic? ...
Yes, it will be the same, but not across library versions (and possibly not
even across architectures).

So possible that in v1.9.0 geth was using an older snappy version than master, causing the difference in compressed receipts that I saw via cmp.

holiman

I need to do some more review of this, but have some comments so far

core/rawdb/freezer.go

holiman · 2021-11-30T13:29:58Z

cmd/geth/dbcmd.go

+		log.Info("No legacy receipts to migrate", "number", firstIdx)
+		return nil
+	}
+	log.Info("First legacy receipt", "number", firstIdx)


Afaict, this is the last time you make any use of firstIdx: the rest of the code starts over from 0 and feeds it into the transformer. And the transformer will (?) return stop=true for the first empty receipt list, so I don't see how this can work?

I had to double-check to see why its working :) so an empty receipt list is both legacy and non-legacy and transformer will ask is this legacy? it is so stop=false. stop will be true only for the first non-empty non-legacy receipt list

So I was planning to pass firstIdx to MigrateTable so we could start from there but we also need to copy over the first empty receipts so I decided to keep MigrateTable as is and only use firstIdx to determine whether the db is in the old format.

holiman · 2021-11-30T13:35:01Z

core/rawdb/table.go

@@ -101,6 +101,10 @@ func (t *table) Sync() error {
 	return t.db.Sync()
 }

+func (t *table) MigrateTable(kind string, fn TransformerFn) error {


holiman · 2021-11-30T13:35:42Z

ethdb/database.go

@@ -111,6 +111,9 @@ type AncientWriter interface {

 	// Sync flushes all in-memory ancient store data to disk.
 	Sync() error
+
+	// MigrateTable Processes and migrates entries in a table to a new format.


Please document the semantics of the func([]byte)([]byte, bool, error)

Co-authored-by: Martin Holst Swende <martin@swende.se>

holiman · 2021-11-30T17:28:10Z

Considering that this will only ever be used by a very very small percentage of users, and only ever once, I think we should prefer simplicity over speed. So maybe let's aim for the cleanest possible way to do it.

…acy-receipt-converter

s1na · 2021-11-30T17:35:17Z

Fair enough, I'll do a re-write which simply copies over all entries.

fjl · 2022-03-09T00:08:19Z

Closing because of #24028

Ragnar00851 approved these changes Sep 6, 2021

View reviewed changes

s1na added 15 commits November 9, 2021 15:30

core/types: add legacy receipt encoding, conversion test

9351cac

core/types: add convertLegacyStoredReceipt method

6bae0aa

core/types: encode list of legacy receipts

3a3ca13

core/types: add method checking if receipt is legacy

b250783

cmd/geth: add boilerplate for freezer-migrate cmd

45b73a3

core/types: add method for converting list of legacy receipts

7474ef0

cmd/geth: write converted receipts to tmp freezer file

39e5aee

cmd,core,ethdb: drop receipts table after copying over new receipts

b1c67fd

cmd,core,ethdb: move logic to transform table fn

82f36c9

core/rawdb: remove getFileBounds

05dade6

core/rawdb: copy over rest of index bits

dce1336

core/rawdb: drop table append

16c55dc

core/rawdb: use new freezer funcs

710b7d9

core/rawdb: forgot batch commit

16286fa

core/rawdb: comment, minor fixes

05812de

s1na force-pushed the legacy-receipt-converter branch from 0f2a255 to 05812de Compare November 15, 2021 12:47

s1na added 9 commits November 16, 2021 16:41

core/rawdb: handle clean switch-over boundary

99c1a32

core/rawdb: fix off-by-one err

66b5285

core/rawdb: add tableFilePath method

8092cdd

cmd,core: rename files after switchover

26ec367

core/rawdb: fix file juggling

fd000bc

core/rawdb: remove DropTable method

e4028b6

ethdb: minor

e3b1daa

core/rawdb: improve readability

0d22403

core/rawdb: fix initial legacy check

8466887

s1na added 5 commits November 17, 2021 21:51

Add start param to migrateTable

eb306b6

check batch append err

76426e3

drop start param from migrateTable

1279bb3

add license to legacy file

7338e21

rm helper funcs and their tests

63fb341

s1na marked this pull request as ready for review November 18, 2021 09:01

s1na requested review from holiman, karalabe and rjl493456442 as code owners November 18, 2021 09:01

ligi assigned holiman Nov 30, 2021

holiman reviewed Nov 30, 2021

View reviewed changes

freezer: rm unneeded line

fa28df5

Co-authored-by: Martin Holst Swende <martin@swende.se>

s1na added 2 commits November 30, 2021 18:29

core, ethdb: add comments

6bef481

Merge remote-tracking branch 's1na/legacy-receipt-converter' into leg…

2b5b633

…acy-receipt-converter

s1na mentioned this pull request Dec 1, 2021

core/rawdb: simple legacy receipt converter #24028

Merged

fjl closed this Mar 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Legacy receipt converter tool #23454

Legacy receipt converter tool #23454

s1na commented Aug 24, 2021

holiman commented Aug 24, 2021 via email

s1na commented Aug 24, 2021 •

edited

holiman commented Aug 24, 2021 via email

MariusVanDerWijden commented Sep 6, 2021 •

edited

s1na commented Nov 18, 2021

s1na commented Nov 19, 2021 •

edited

holiman left a comment

holiman Nov 30, 2021

s1na Nov 30, 2021

s1na Nov 30, 2021

holiman Nov 30, 2021

holiman Nov 30, 2021

holiman commented Nov 30, 2021 via email

s1na commented Nov 30, 2021

fjl commented Mar 9, 2022

Legacy receipt converter tool #23454

Legacy receipt converter tool #23454

Conversation

s1na commented Aug 24, 2021

holiman commented Aug 24, 2021 via email

s1na commented Aug 24, 2021 • edited

holiman commented Aug 24, 2021 via email

MariusVanDerWijden commented Sep 6, 2021 • edited

s1na commented Nov 18, 2021

s1na commented Nov 19, 2021 • edited

holiman left a comment

Choose a reason for hiding this comment

holiman Nov 30, 2021

Choose a reason for hiding this comment

s1na Nov 30, 2021

Choose a reason for hiding this comment

s1na Nov 30, 2021

Choose a reason for hiding this comment

holiman Nov 30, 2021

Choose a reason for hiding this comment

holiman Nov 30, 2021

Choose a reason for hiding this comment

holiman commented Nov 30, 2021 via email

s1na commented Nov 30, 2021

fjl commented Mar 9, 2022

s1na commented Aug 24, 2021 •

edited

MariusVanDerWijden commented Sep 6, 2021 •

edited

s1na commented Nov 19, 2021 •

edited