Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

miner: fix data race #23435

Merged
merged 4 commits into from Oct 8, 2021
Merged

miner: fix data race #23435

merged 4 commits into from Oct 8, 2021

Conversation

MariusVanDerWijden
Copy link
Member

Fixes a race between close and makeCurrent

==================
WARNING: DATA RACE
Read at 0x00c00048c5a8 by goroutine 128:
  github.com/ethereum/go-ethereum/miner.(*worker).close()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:321 +0x4a
  github.com/ethereum/go-ethereum/miner.(*Miner).update()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:141 +0xa88

Previous write at 0x00c00048c5a8 by goroutine 124:
  github.com/ethereum/go-ethereum/miner.(*worker).makeCurrent()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:695 +0x8d4
  github.com/ethereum/go-ethereum/miner.(*worker).commitNewWork()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:938 +0x818
  github.com/ethereum/go-ethereum/miner.(*worker).mainLoop()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:456 +0x58d

Goroutine 128 (running) created at:
  github.com/ethereum/go-ethereum/miner.New()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:78 +0x30c
  github.com/ethereum/go-ethereum/miner.createMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:260 +0x84f
  github.com/ethereum/go-ethereum/miner.TestCloseMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:185 +0x3c
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Goroutine 124 (running) created at:
  github.com/ethereum/go-ethereum/miner.newWorker()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:228 +0xa52
  github.com/ethereum/go-ethereum/miner.New()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:76 +0x128
  github.com/ethereum/go-ethereum/miner.createMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:260 +0x84f
  github.com/ethereum/go-ethereum/miner.TestCloseMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:185 +0x3c
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202
==================
==================
WARNING: DATA RACE
Read at 0x00c0002d8a30 by goroutine 128:
  github.com/ethereum/go-ethereum/miner.(*worker).close()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:321 +0x19e
  github.com/ethereum/go-ethereum/miner.(*Miner).update()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:141 +0xa88

Previous write at 0x00c0002d8a30 by goroutine 124:
  github.com/ethereum/go-ethereum/miner.(*worker).makeCurrent()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:671 +0x31c
  github.com/ethereum/go-ethereum/miner.(*worker).commitNewWork()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:938 +0x818
  github.com/ethereum/go-ethereum/miner.(*worker).mainLoop()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:456 +0x58d

Goroutine 128 (running) created at:
  github.com/ethereum/go-ethereum/miner.New()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:78 +0x30c
  github.com/ethereum/go-ethereum/miner.createMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:260 +0x84f
  github.com/ethereum/go-ethereum/miner.TestCloseMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:185 +0x3c
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Goroutine 124 (running) created at:
  github.com/ethereum/go-ethereum/miner.newWorker()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:228 +0xa52
  github.com/ethereum/go-ethereum/miner.New()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:76 +0x128
  github.com/ethereum/go-ethereum/miner.createMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:260 +0x84f
  github.com/ethereum/go-ethereum/miner.TestCloseMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:185 +0x3c
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202
==================
==================
WARNING: DATA RACE
Read at 0x00c000215690 by goroutine 128:
  github.com/ethereum/go-ethereum/core/state.(*StateDB).StopPrefetcher()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/state/statedb.go:168 +0x106
  github.com/ethereum/go-ethereum/miner.(*worker).close()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:322 +0xb5
  github.com/ethereum/go-ethereum/miner.(*Miner).update()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:141 +0xa88

Previous write at 0x00c000215690 by goroutine 124:
  github.com/ethereum/go-ethereum/core/state.New()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/state/statedb.go:128 +0x384
  github.com/ethereum/go-ethereum/core.(*BlockChain).StateAt()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/blockchain.go:691 +0x19a
  github.com/ethereum/go-ethereum/miner.(*worker).makeCurrent()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:665 +0x71
  github.com/ethereum/go-ethereum/miner.(*worker).commitNewWork()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:938 +0x818
  github.com/ethereum/go-ethereum/miner.(*worker).mainLoop()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:456 +0x58d

Goroutine 128 (running) created at:
  github.com/ethereum/go-ethereum/miner.New()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:78 +0x30c
  github.com/ethereum/go-ethereum/miner.createMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:260 +0x84f
  github.com/ethereum/go-ethereum/miner.TestCloseMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:185 +0x3c
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Goroutine 124 (running) created at:
  github.com/ethereum/go-ethereum/miner.newWorker()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:228 +0xa52
  github.com/ethereum/go-ethereum/miner.New()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:76 +0x128
  github.com/ethereum/go-ethereum/miner.createMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:260 +0x84f
  github.com/ethereum/go-ethereum/miner.TestCloseMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:185 +0x3c
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202
==================
==================
WARNING: DATA RACE
Read at 0x00c00010dd78 by goroutine 128:
  github.com/ethereum/go-ethereum/core/state.(*triePrefetcher).close()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/state/trie_prefetcher.go:78 +0x64
  github.com/ethereum/go-ethereum/core/state.(*StateDB).StopPrefetcher()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/state/statedb.go:169 +0x13c
  github.com/ethereum/go-ethereum/miner.(*worker).close()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:322 +0xb5
  github.com/ethereum/go-ethereum/miner.(*Miner).update()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:141 +0xa88

Previous write at 0x00c00010dd78 by goroutine 124:
  github.com/ethereum/go-ethereum/core/state.newTriePrefetcher()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/state/trie_prefetcher.go:57 +0x912
  github.com/ethereum/go-ethereum/core/state.(*StateDB).StartPrefetcher()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/state/statedb.go:161 +0x147
  github.com/ethereum/go-ethereum/miner.(*worker).makeCurrent()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:669 +0x1db
  github.com/ethereum/go-ethereum/miner.(*worker).commitNewWork()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:938 +0x818
  github.com/ethereum/go-ethereum/miner.(*worker).mainLoop()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:456 +0x58d

Goroutine 128 (running) created at:
  github.com/ethereum/go-ethereum/miner.New()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:78 +0x30c
  github.com/ethereum/go-ethereum/miner.createMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:260 +0x84f
  github.com/ethereum/go-ethereum/miner.TestCloseMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:185 +0x3c
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Goroutine 124 (running) created at:
  github.com/ethereum/go-ethereum/miner.newWorker()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/worker.go:228 +0xa52
  github.com/ethereum/go-ethereum/miner.New()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner.go:76 +0x128
  github.com/ethereum/go-ethereum/miner.createMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:260 +0x84f
  github.com/ethereum/go-ethereum/miner.TestCloseMiner()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/miner/miner_test.go:185 +0x3c
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202
==================
--- FAIL: TestCloseMiner (0.04s)
    testing.go:1092: race detected during execution of test

@MariusVanDerWijden
Copy link
Member Author

The second commit fixes a datarace between different go routines that all want to update the syncChallengeTimeout. By not allowing them to run in parallel we enforce the atomicity of updating it.
Running them in parallel only saves ~4s (out of 17s) on my 24 core machine

eth/eth_handler_test.go
WARNING: DATA RACE
Read at 0x000001647448 by goroutine 81:
  github.com/ethereum/go-ethereum/eth.testCheckpointChallenge()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/eth/handler_eth_test.go:496 +0x65
  github.com/ethereum/go-ethereum/eth.TestCheckpointChallenge.func1()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/eth/handler_eth_test.go:487 +0x113
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Previous write at 0x000001647448 by goroutine 76:
  github.com/ethereum/go-ethereum/eth.testCheckpointChallenge()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/eth/handler_eth_test.go:497 +0xbc
  github.com/ethereum/go-ethereum/eth.TestCheckpointChallenge.func1()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/eth/handler_eth_test.go:487 +0x113
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

@MariusVanDerWijden
Copy link
Member Author

The third commit fixes a race in feehistory_test where the caching go-routines read the london block before it was actually written:

WARNING: DATA RACE
Write at 0x000000f1c3c8 by goroutine 33:
  github.com/ethereum/go-ethereum/eth/gasprice.newTestBackend()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/eth/gasprice/gasprice_test.go:104 +0x304
  github.com/ethereum/go-ethereum/eth/gasprice.TestFeeHistory()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/eth/gasprice/feehistory_test.go:59 +0x7d2
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Previous read at 0x000000f1c3c8 by goroutine 35:
  github.com/ethereum/go-ethereum/params.(*ChainConfig).IsLondon()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/params/config.go:432 +0x2fe
  github.com/ethereum/go-ethereum/params.(*ChainConfig).Rules()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/params/config.go:636 +0x32b
  github.com/ethereum/go-ethereum/core/vm.NewEVM()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/vm/evm.go:134 +0x76
  github.com/ethereum/go-ethereum/core.(*statePrefetcher).Prefetch()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/state_prefetcher.go:55 +0x364
  github.com/ethereum/go-ethereum/core.(*BlockChain).insertChain.func3()
      /home/matematik/go/src/github.com/ethereum/go-ethereum/core/blockchain.go:1852 +0x145
      ```

@holiman
Copy link
Contributor

holiman commented Aug 23, 2021

LGTM so far, but maybe let's wait a few days with merging

@rjl493456442
Copy link
Member

The miner fix looks good to me.

miner/worker.go Outdated
@@ -318,6 +318,8 @@ func (w *worker) isRunning() bool {
// close terminates all background threads maintained by the worker.
// Note the worker does not support being closed multiple times.
func (w *worker) close() {
w.mu.Lock()
defer w.mu.Unlock()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, I think one better fix is to move the w.current.state.StopPrefetcher() operation into the mainLoop. Since w.current is only available in this loop and you can tear down it in the defer function if the worker is stopped.

Copy link
Contributor

@holiman holiman Sep 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into this a bit more...
The worker.close is only called when exitCh is closed, which happens through miner.Close (well, aside from various tests that also use it).

I renamed it like this:

func (miner *Miner) zClose() {
	close(miner.exitCh)
}

And geth still compiles. So I'm wondering, do we actually ever call miner.Close() ->close(exitCh) -> worker.close() ?

@holiman
Copy link
Contributor

holiman commented Aug 27, 2021

The last change slightly changes the sematics of close.

  • previously, whenever close returned, the prefetcher was guaranteed stopped.
  • with this change, close can return before the prefetcher is fully stopped, since it only closes a chan but does not wait for the mainloop to exit.

I don't whether this change is important or not, but wanted to highlight it so we can figure out whether it matters or not.

@holiman
Copy link
Contributor

holiman commented Aug 31, 2021

The semantics-change is probably fine, if this only ever happens during shutdown.

Copy link
Contributor

@holiman holiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@holiman
Copy link
Contributor

holiman commented Sep 8, 2021

Actually, there's one more semantic change. This change will cause the prefetcher to be stopped on worker errors:

		case <-w.exitCh:
			return
		case <-w.txsSub.Err():
			return
		case <-w.chainHeadSub.Err():
			return
		case <-w.chainSideSub.Err():
			return

Is that the correct thing to do? Previously it would only happen in conjunction with w.exitCh being triggered.

Copy link
Contributor

@fjl fjl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested this a bit, and it seems to work fine.

@holiman re

So I'm wondering, do we actually ever call miner.Close() ->close(exitCh) -> worker.close() ?

This is now being called because you changed .Stop() to .Close() in #21992.

@fjl fjl merged commit ee120ef into ethereum:master Oct 8, 2021
@fjl fjl added this to the 1.10.10 milestone Oct 8, 2021
sidhujag pushed a commit to syscoin/go-ethereum that referenced this pull request Oct 9, 2021
This fixes a data race on worker.current by moving the call to StopPrefetcher
into the main loop.

The commit also contains fixes for two other races in unit tests of unrelated packages.
@MariusVanDerWijden MariusVanDerWijden deleted the miner-race branch November 30, 2021 15:26
yongjun925 pushed a commit to DODOEX/go-ethereum that referenced this pull request Dec 3, 2022
This fixes a data race on worker.current by moving the call to StopPrefetcher
into the main loop.

The commit also contains fixes for two other races in unit tests of unrelated packages.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants