Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

concurrent map read and map write #22892

Closed
unclezoro opened this issue May 18, 2021 · 5 comments
Closed

concurrent map read and map write #22892

unclezoro opened this issue May 18, 2021 · 5 comments
Labels
type:bug

Comments

@unclezoro
Copy link

System information

Geth version: 1.10.3
OS & Version: Linux
Commit hash : (if develop)

Expected behaviour

geth can gracefully shutdown

Actual behaviour

geth crash when reciev TERM signal

The error log is:

fatal error: concurrent map read and map write

goroutine 1174638587 [running]:
runtime.throw(0x16a9e0f, 0x21)
        runtime/panic.go:1117 +0x72 fp=0xc03d0f3e98 sp=0xc03d0f3e68 pc=0x44f1f2
runtime.mapaccess2(0x148d5a0, 0xc01bcef680, 0xc03d0f3f30, 0x27c86a0, 0x1af9c00)
        runtime/map.go:469 +0x255 fp=0xc03d0f3ed8 sp=0xc03d0f3e98 pc=0x4262d5
github.com/ethereum/go-ethereum/trie.(*Database).commit(0xc015e32000, 0xd896a55ad5f3378c, 0xb8762c019361b270, 0x9a39c7b46f260255, 0x5a85e51f6b6c9ef5, 0x1ae8098, 0xc258749230, 0xc4c43b9090, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum/trie/database.go:764 +0x74 fp=0xc03d0f3fc0 sp=0xc03d0f3ed8 pc=0x7c8d74
github.com/ethereum/go-ethereum/trie.(*Database).commit.func1(0xd896a55ad5f3378c, 0xb8762c019361b270, 0x9a39c7b46f260255, 0x5a85e51f6b6c9ef5)        github.com/ethereum/go-ethereum/trie/database.go:771 +0x85 fp=0xc03d0f4030 sp=0xc03d0f3fc0 pc=0x7e3ea5
github.com/ethereum/go-ethereum/trie.forGatherChildren(0x1ad0468, 0xc36bfafcf8, 0xc03d0f44a0)
        github.com/ethereum/go-ethereum/trie/database.go:206 +0x258 fp=0xc03d0f41c0 sp=0xc03d0f4030 pc=0x7c3df8
github.com/ethereum/go-ethereum/trie.forGatherChildren(0x1ad0490, 0xc13fd3d560, 0xc03d0f44a0)
        github.com/ethereum/go-ethereum/trie/database.go:203 +0xe5 fp=0xc03d0f4350 sp=0xc03d0f41c0 pc=0x7c3c85
github.com/ethereum/go-ethereum/trie.(*cachedNode).forChilds(0xc269a7ff80, 0xc03d0f44a0)
        github.com/ethereum/go-ethereum/trie/database.go:191 +0x110 fp=0xc03d0f4400 sp=0xc03d0f4350 pc=0x7c3b90
github.com/ethereum/go-ethereum/trie.(*Database).commit(0xc015e32000, 0x32bbdd73f4db33b6, 0x267ba816a65a56d6, 0x48f0e6cb047b9bbb, 0xb2fb675c2064d90f, 0x1ae8098, 0xc258749230, 0xc4c43b9090, 0x0, 0x1ad0468, ...)

Steps to reproduce the behaviour

Backtrace

[backtrace]

When submitting logs: please submit them as text and not screenshots.
@karalabe
Copy link
Member

You'll need to send us more of that crash log. It is missing the part with the second goroutine that does the concurrent read/write

@unclezoro
Copy link
Author

trace.log
Hi @karalabe above the whole goroutine

@holiman
Copy link
Contributor

holiman commented May 20, 2021

As far as I can tell from the stacktrace, it's the miner competing with the shutdown:

fatal error: concurrent map read and map write

goroutine 1192435606 [running]:
runtime.throw(0x16a9e0f, 0x21)
        runtime/panic.go:1117 +0x72 fp=0xc03a92e3c0 sp=0xc03a92e390 pc=0x44f1f2
runtime.mapaccess2(0x148d5a0, 0xc000adb5c0, 0xc03a92e458, 0x3e21e84d1a0679a, 0xc2d85c33b32a604d)
        runtime/map.go:469 +0x255 fp=0xc03a92e400 sp=0xc03a92e3c0 pc=0x4262d5
github.com/ethereum/go-ethereum/trie.(*Database).commit(0xc0008ba5b0, 0x36c71ec4d0cdf456, 0xe5aa6ab638521ae, 0x16ac6173d1baa2a1, 0x3f35e773deb67465, 0x1ae8098, 0xc10b24e0a8, 0xc138f7b530, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum/trie/database.go:764 +0x74 fp=0xc03a92e4e8 sp=0xc03a92e400 pc=0x7c8d74
github.com/ethereum/go-ethereum/trie.(*Database).commit.func1(0x36c71ec4d0cdf456, 0xe5aa6ab638521ae, 0x16ac6173d1baa2a1, 0x3f35e773deb67465)
        github.com/ethereum/go-ethereum/trie/database.go:771 +0x85 fp=0xc03a92e558 sp=0xc03a92e4e8 pc=0x7e3ea5
github.com/ethereum/go-ethereum/trie.forGatherChildren(0x1ad0468, 0xc0fa75c9f0, 0xc03a92e9c8)
        github.com/ethereum/go-ethereum/trie/database.go:206 +0x258 fp=0xc03a92e6e8 sp=0xc03a92e558 pc=0x7c3df8
github.com/ethereum/go-ethereum/trie.forGatherChildren(0x1ad0490, 0xc0f5218c60, 0xc03a92e9c8)
        github.com/ethereum/go-ethereum/trie/database.go:203 +0xe5 fp=0xc03a92e878 sp=0xc03a92e6e8 pc=0x7c3c85
github.com/ethereum/go-ethereum/trie.(*cachedNode).forChilds(0xc0ae1d81e0, 0xc03a92e9c8)
        github.com/ethereum/go-ethereum/trie/database.go:191 +0x110 fp=0xc03a92e928 sp=0xc03a92e878 pc=0x7c3b90
...
github.com/ethereum/go-ethereum/trie.(*Database).Commit(0xc0008ba5b0, 0x5952d250d0408019, 0x54d395790b8ac578, 0x3251634338a85d57, 0xe0d71cdd56a2448b, 0x1, 0x0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/database.go:723 +0x172 fp=0xc03a931bc8 sp=0xc03a931a80 pc=0x7c8172
github.com/ethereum/go-ethereum/core.(*BlockChain).Stop(0xc0000cb900)
        github.com/ethereum/go-ethereum/core/blockchain.go:1027 +0x625 fp=0xc03a931e70 sp=0xc03a931bc8 pc=0xdb1425
github.com/ethereum/go-ethereum/eth.(*Ethereum).Stop(0xc01d23eb40, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/backend.go:577 +0xd7 fp=0xc03a931ea8 sp=0xc03a931e70 pc=0x1046c77
github.com/ethereum/go-ethereum/node.(*Node).stopServices(0xc0000aed00, 0xc13747e260, 0x2, 0x2, 0x100000000000000, 0xf5a)
        github.com/ethereum/go-ethereum/node/node.go:303 +0xc2 fp=0xc03a931f28 sp=0xc03a931ea8 pc=0x1002902
github.com/ethereum/go-ethereum/node.(*Node).Close(0xc0000aed00, 0x0, 0x0)
        github.com/ethereum/go-ethereum/node/node.go:223 +0x145 fp=0xc03a931fc8 sp=0xc03a931f28 pc=0x1001d05
runtime.goexit()
        runtime/asm_amd64.s:1371 +0x1 fp=0xc03a931fd0 sp=0xc03a931fc8 pc=0x4892e1
created by github.com/ethereum/go-ethereum/cmd/utils.StartNode.func1
        github.com/ethereum/go-ethereum/cmd/utils/cmd.go:90 +0x1f3



goroutine 233 [runnable]:
github.com/ethereum/go-ethereum/trie.(*Database).insert.func1(0x89d3a7b3af251199, 0x24e755fc0afe4a05, 0x1fb35880ecdfe931, 0x573bea56c478b85d)
        github.com/ethereum/go-ethereum/trie/database.go:336 +0x77
github.com/ethereum/go-ethereum/trie.forGatherChildren(0x1ad0468, 0xc207924540, 0xc2366ed860)
        github.com/ethereum/go-ethereum/trie/database.go:206 +0x258
github.com/ethereum/go-ethereum/trie.forGatherChildren(0x1ad0490, 0xc1a9b19680, 0xc2366ed860)
        github.com/ethereum/go-ethereum/trie/database.go:203 +0xe5
github.com/ethereum/go-ethereum/trie.(*cachedNode).forChilds(0xc17da28d80, 0xc2366ed860)
        github.com/ethereum/go-ethereum/trie/database.go:191 +0x110
github.com/ethereum/go-ethereum/trie.(*Database).insert(0xc0008ba5b0, 0xdbd643f373d3d05, 0x10e36faced1ef651, 0x4402ad7a700d5e3f, 0xab1e28d3ba7bd6cb, 0x73, 0x1ace7f8, 0xc0b63912c0)
        github.com/ethereum/go-ethereum/trie/database.go:336 +0x150
github.com/ethereum/go-ethereum/trie.(*committer).store(0xc1c2a48ec0, 0x1ace7f8, 0xc0b63912c0, 0xc0008ba5b0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/committer.go:200 +0x386
github.com/ethereum/go-ethereum/trie.(*committer).commit(0xc1c2a48ec0, 0x1ace7f8, 0xc086ee4000, 0xc0008ba5b0, 0xc000b00400, 0x0, 0xc2366edd78, 0x4220f9)
        github.com/ethereum/go-ethereum/trie/committer.go:124 +0x54a
github.com/ethereum/go-ethereum/trie.(*committer).commitChildren(0xc1c2a48ec0, 0xc086eb5cc0, 0xc0008ba5b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum/trie/committer.go:155 +0x137
github.com/ethereum/go-ethereum/trie.(*committer).commit(0xc1c2a48ec0, 0x1ace7f8, 0xc086eb5cc0, 0xc0008ba5b0, 0xc000b00400, 0x0, 0xc2366ee280, 0x4220f9)
        github.com/ethereum/go-ethereum/trie/committer.go:117 +0x41e
github.com/ethereum/go-ethereum/trie.(*committer).commitChildren(0xc1c2a48ec0, 0xc086eb5a40, 0xc0008ba5b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum/trie/committer.go:155 +0x137
github.com/ethereum/go-ethereum/trie.(*committer).commit(0xc1c2a48ec0, 0x1ace7f8, 0xc086eb5a40, 0xc0008ba5b0, 0xc000b00400, 0x0, 0xc2366ee788, 0x4220f9)
        github.com/ethereum/go-ethereum/trie/committer.go:117 +0x41e
github.com/ethereum/go-ethereum/trie.(*committer).commitChildren(0xc1c2a48ec0, 0xc086eb57c0, 0xc0008ba5b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum/trie/committer.go:155 +0x137
github.com/ethereum/go-ethereum/trie.(*committer).commit(0xc1c2a48ec0, 0x1ace7f8, 0xc086eb57c0, 0xc0008ba5b0, 0x1ad0468, 0xc33999f488, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/committer.go:117 +0x41e
github.com/ethereum/go-ethereum/trie.(*committer).commitChildren(0xc1c2a48ec0, 0xc086eac000, 0xc0008ba5b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum/trie/committer.go:155 +0x137
github.com/ethereum/go-ethereum/trie.(*committer).commit(0xc1c2a48ec0, 0x1ace7f8, 0xc086eac000, 0xc0008ba5b0, 0xc, 0x1521080, 0xb57f5cf7f5708f01, 0xc216ec6710)
        github.com/ethereum/go-ethereum/trie/committer.go:117 +0x41e
github.com/ethereum/go-ethereum/trie.(*committer).Commit(0xc1c2a48ec0, 0x1ace7f8, 0xc086eac000, 0xc0008ba5b0, 0x6f4eb8ab191e3201, 0x7f6d31da8a68, 0x30, 0xc271ed5bf0, 0x0)
        github.com/ethereum/go-ethereum/trie/committer.go:80 +0x56
github.com/ethereum/go-ethereum/trie.(*Trie).Commit(0xc138c50b40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/trie.go:539 +0x22f
github.com/ethereum/go-ethereum/trie.(*SecureTrie).Commit(0xc138c50b40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/secure_trie.go:160 +0x389
github.com/ethereum/go-ethereum/core/state.(*stateObject).CommitTrie(0xc1e858c780, 0x1ae7ee8, 0xc000b6f260, 0x0, 0x0)
        github.com/ethereum/go-ethereum/core/state/state_object.go:411 +0xe9
github.com/ethereum/go-ethereum/core/state.(*StateDB).Commit(0xc138cc64e0, 0xc2366efa01, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/core/state/statedb.go:930 +0x37e
github.com/ethereum/go-ethereum/core.(*BlockChain).writeBlockWithState(0xc0000cb900, 0xc2170fe360, 0xc06c252000, 0xd2, 0xd2, 0xc003ca6a80, 0x2dc, 0x550, 0xc138cc64e0, 0xc000b00401, ...)
        github.com/ethereum/go-ethereum/core/blockchain.go:1480 +0x76c
github.com/ethereum/go-ethereum/core.(*BlockChain).WriteBlockWithState(0xc0000cb900, 0xc2170fe360, 0xc06c252000, 0xd2, 0xd2, 0xc003ca6a80, 0x2dc, 0x550, 0xc138cc64e0, 0x1, ...)
        github.com/ethereum/go-ethereum/core/blockchain.go:1448 +0x111
github.com/ethereum/go-ethereum/miner.(*worker).resultLoop(0xc0ac9b1b00)
        github.com/ethereum/go-ethereum/miner/worker.go:641 +0x80a
created by github.com/ethereum/go-ethereum/miner.newWorker
        github.com/ethereum/go-ethereum/miner/worker.go:236 +0x585

@holiman
Copy link
Contributor

holiman commented May 20, 2021

This might be fixed by #22853

@piersy piersy mentioned this issue Oct 12, 2021
Closed
@holiman
Copy link
Contributor

holiman commented Oct 20, 2021

Fixed by #22853

@holiman holiman closed this as completed Oct 20, 2021
@kyrie-yl kyrie-yl mentioned this issue Dec 16, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ligi @karalabe @holiman @rjl493456442 @unclezoro