core/rawdb, ethdb: introduce batched/atomic reads from ancients #23566

holiman · 2021-09-13T09:12:00Z

In several places, we use the following mechanic:

Check if something exists in freezer,
If not, check if it exists in leveldb,
If not, check again if it exists in freezer,

This is because between 1 and 2, there's a (ever so slight) chance that it was written into ancients and deleted from leveldb.

As of #23462, we now do those types of transfers using ancient batches, and thus introduced the ability to do locked writes. This PR extends that ability to support locked reads.

This could be used in the following situations, and both simplify and reduce the code:

core/rawdb/accessors_chain.go

holiman · 2021-09-13T10:20:09Z

From a design-perspective, maybe the interfaces should look differently. This PR so far just adds AtomicReadAncients(fn func(AncientReader) error) (err error) to the AncientReader interface -- which adds an interface recursion in theory, but which doesn't hold in practice.

So theoretically, this PR allows to do this:

	db.AtomicReadAncients(func(reader ethdb.AncientReader) error {
		reader.AtomicReadAncients(func(reader ethdb.AncientReader) error {
			reader.AncientSize("foo")
		})
	})

In practice, that would lead to a double-RLock(). So maybe there should be an AncientReader with the base operations, and an AncientAtomicReader which is AncientReader + AtomicReadAncients ?

Or maybe it should be something else entirely.

holiman · 2021-09-16T10:24:14Z

In practice, that would lead to a double-RLock(). So maybe there should be an AncientReader with the base operations, and an AncientAtomicReader which is AncientReader + AtomicReadAncients ?

This is now fixed.

ethdb/database.go

holiman · 2021-09-20T10:56:18Z

core/rawdb/accessors_chain.go

+	var data []byte
+	db.AtomicReadAncients(func(reader ethdb.AncientReader) error {
+		// Check if the data is in ancients
+		if h, err := reader.Ancient(freezerHashTable, number); err == nil && common.BytesToHash(h) == hash {


use bytes.Equal instead of BytesToHash

core/rawdb/accessors_chain.go

fjl

Looks really nice so far!

rjl493456442

Have we ever run the fast/snaps sync with this PR? My concern is that previously ancient writes and reads can happen concurrently but now it's not allowed. Ancient read is pretty common during the sync. Not sure how much performance will be affected.

rjl493456442 · 2021-09-28T10:58:26Z

ethdb/database.go

@@ -90,6 +90,14 @@ type AncientReader interface {
 	AncientSize(kind string) (uint64, error)
 }

+// AncientBatchReader is the interface for 'batched' or 'atomic' reading.
+type AncientBatchReader interface {
+	AncientReader


Please add one more line here.

AncientBatchReader looks confusing. Maybe we can rename the AncientReader to RawAncientReader and rename the AncientBatchReader to AncientReader.

core/rawdb/accessors_chain.go

rjl493456442 · 2021-09-28T11:35:46Z

ethdb/database.go

@@ -90,6 +90,14 @@ type AncientReader interface {
 	AncientSize(kind string) (uint64, error)
 }

+// AncientBatchReader is the interface for 'batched' or 'atomic' reading.
+type AncientBatchReader interface {
+	AncientReader


AncientBatchReader looks confusing. Maybe we can rename the AncientReader to RawAncientReader and rename the AncientBatchReader to AncientReader.

rjl493456442 · 2021-09-28T11:36:25Z

ethdb/database.go

+	AncientReader
+	// ReadAncients runs the given read operation while ensuring that no writes take place
+	// on the underlying freezer.
+	ReadAncients(fn func(AncientReader) error) (err error)


Perhaps ReadAncientsLocked?

The ReadAncients operation is analoguous to ModifyAncients which is the locked/atomic/batched write operation.

holiman · 2021-09-28T11:48:37Z

Have we ever run the fast/snaps sync with this PR? My concern is that previously ancient writes and reads can happen concurrently but now it's not allowed

Ancient writes are already non-concurrent. But yes, you might have a point regarding the reads.

I've done snap syncs with PR's on top of this one, but I haven't run a head-to-head benchmark. I'll spin one up

holiman · 2021-09-28T12:08:37Z

This PR running on bench03 versus master on bench04, https://geth-bench.ethdevops.io/d/Jpk-Be5Wk/dual-geth?orgId=1&from=1632830932283&to=now

holiman · 2021-09-28T13:31:37Z

looks fine (pr is yellow)

holiman · 2021-09-29T17:07:04Z

The syncs were pretty much same-same

core/rawdb/accessors_chain.go

fjl

We had a review call about this, and here are my thoughts: This PR is good because it removes the complexity of having to think about concurrent modifications of the freezer while reading it.

The atomic read possibility (ReadAncients) is added here without removing the previous non-atomic freezer API. The practical difference between these two ways of accessing the freezer is subtle and related to freezer internals. I think that's not great because it increases the complexity of the API. But keeping non-atomic access as a possibility is the correct choice: we should still be able to write ancient chain data while other peers read headers.

This reverts commit edc5a92.

…reum#23566) This PR adds a new accessor method to the freezer database. This new view offers a consistent interface, guaranteeing that all individual tables (headers, bodies etc) are all on the same number, and that this number is not changes (added/truncated) while the operation is performing.

holiman requested review from karalabe and rjl493456442 as code owners September 13, 2021 09:12

holiman added the status:triage label Sep 13, 2021

holiman force-pushed the atomic_ancients branch from 0143295 to ebed66d Compare September 13, 2021 09:57

MariusVanDerWijden reviewed Sep 13, 2021

View reviewed changes

core/rawdb/accessors_chain.go Outdated Show resolved Hide resolved

holiman commented Sep 20, 2021

View reviewed changes

ethdb/database.go Outdated Show resolved Hide resolved

holiman commented Sep 20, 2021

View reviewed changes

ethdb/database.go Show resolved Hide resolved

holiman commented Sep 20, 2021

View reviewed changes

fjl reviewed Sep 20, 2021

View reviewed changes

core/rawdb/accessors_chain.go Outdated Show resolved Hide resolved

fjl reviewed Sep 20, 2021

View reviewed changes

holiman force-pushed the atomic_ancients branch 2 times, most recently from de88963 to 2a0c9dd Compare September 20, 2021 20:16

holiman mentioned this pull request Sep 20, 2021

core/rawdb: use AncientRange when initializing leveldb from freezer #23612

Merged

holiman removed the status:triage label Sep 21, 2021

holiman mentioned this pull request Sep 21, 2021

core, eth: improve delivery speed on header requests #23105

Merged

holiman requested a review from fjl September 28, 2021 08:14

rjl493456442 reviewed Sep 28, 2021

View reviewed changes

holiman mentioned this pull request Oct 12, 2021

core: generic test suite for AncientStore #23707

Open

MichaelMure reviewed Oct 12, 2021

View reviewed changes

core/rawdb/accessors_chain.go Show resolved Hide resolved

fjl approved these changes Oct 25, 2021

View reviewed changes

holiman added this to the 1.10.12 milestone Oct 25, 2021

holiman added 2 commits October 25, 2021 14:36

core/rawdb, ethdb: introduce batched/atomic reads from ancients

f2167e3

core/rawdb: use atomic reads

963069c

holiman added 11 commits October 25, 2021 14:36

core/rawdb: minor fixes

afb636a

core/rawdb: check limit before reading data

118eaaa

ethdb: restructure the ancientreader api

8cf75c2

Revert "core/rawdb: check limit before reading data"

677cc9a

This reverts commit edc5a92.

core/rawdb: minor nit

b649011

core/rawdb: fix nofreezedb + errors

84eca45

core/rawdb: fix reading with 0-hash

55c6775

core/rawdb: simplify condition check

bba127b

core/rawdb, ethdb: rename ReadAncients -> AncientRange

e6facc7

core/rawdb, ethdb: rename AtomicReadAncients -> ReadAncients

2c9461b

ethdb,core/rawdb: fix review comments

797044b

holiman force-pushed the atomic_ancients branch from d1c39e4 to 797044b Compare October 25, 2021 12:37

holiman merged commit 0e7efd6 into ethereum:master Oct 25, 2021

holiman deleted the atomic_ancients branch November 10, 2021 18:32

This was referenced Sep 23, 2022

Metadium to master METADIUM/go-metadium#24

Closed

Metadium to master METADIUM/go-metadium#25

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core/rawdb, ethdb: introduce batched/atomic reads from ancients #23566

core/rawdb, ethdb: introduce batched/atomic reads from ancients #23566

holiman commented Sep 13, 2021 •

edited

holiman commented Sep 13, 2021 •

edited

holiman commented Sep 16, 2021

holiman Sep 20, 2021

fjl left a comment

rjl493456442 left a comment

rjl493456442 Sep 28, 2021

rjl493456442 Sep 28, 2021

rjl493456442 Sep 28, 2021

rjl493456442 Sep 28, 2021

holiman Sep 28, 2021

holiman commented Sep 28, 2021

holiman commented Sep 28, 2021 •

edited

holiman commented Sep 28, 2021 •

edited

holiman commented Sep 29, 2021

fjl left a comment •

edited

core/rawdb, ethdb: introduce batched/atomic reads from ancients #23566

core/rawdb, ethdb: introduce batched/atomic reads from ancients #23566

Conversation

holiman commented Sep 13, 2021 • edited

holiman commented Sep 13, 2021 • edited

holiman commented Sep 16, 2021

holiman Sep 20, 2021

Choose a reason for hiding this comment

fjl left a comment

Choose a reason for hiding this comment

rjl493456442 left a comment

Choose a reason for hiding this comment

rjl493456442 Sep 28, 2021

Choose a reason for hiding this comment

rjl493456442 Sep 28, 2021

Choose a reason for hiding this comment

rjl493456442 Sep 28, 2021

Choose a reason for hiding this comment

rjl493456442 Sep 28, 2021

Choose a reason for hiding this comment

holiman Sep 28, 2021

Choose a reason for hiding this comment

holiman commented Sep 28, 2021

holiman commented Sep 28, 2021 • edited

holiman commented Sep 28, 2021 • edited

holiman commented Sep 29, 2021

fjl left a comment • edited

Choose a reason for hiding this comment

holiman commented Sep 13, 2021 •

edited

holiman commented Sep 13, 2021 •

edited

holiman commented Sep 28, 2021 •

edited

holiman commented Sep 28, 2021 •

edited

fjl left a comment •

edited