inspect: add inspect mode for debugging crashed tendermint node #6785

williambanfield · 2021-07-29T23:47:50Z

This change adds a sketch of the Debug mode.

This change adds a Debug struct to the node package. This Debug struct is intended to be created and started by a command in the cmd directory. The Debug struct runs the RPC server on the data directories: both the state store and the block store.

This change required a good deal of refactoring. Namely, a new rpc.go file was added to the node package. This file encapsulates functions for starting RPC servers used by nodes. A potential additional change is to further factor this code into shared code in the rpc package.

Minor API tweaks were also made that seemed appropriate such as the mechanism for fetching routes from the rpc/core package.

Additional work is required to register the Debug service as a command in the cmd directory but I am looking for feedback on if this direction seems appropriate before diving much further.

closes: #5908

rpc/jsonrpc/server/http_server.go

node/setup.go

rpc/core/routes.go

codecov · 2021-07-30T00:06:07Z

Codecov Report

Merging #6785 (3a858ca) into master (d7c3a8f) will decrease coverage by 0.21%.
The diff coverage is 51.30%.

@@            Coverage Diff             @@
##           master    #6785      +/-   ##
==========================================
- Coverage   62.88%   62.66%   -0.22%     
==========================================
  Files         307      309       +2     
  Lines       40464    40564     +100     
==========================================
- Hits        25447    25421      -26     
- Misses      13231    13343     +112     
- Partials     1786     1800      +14

Impacted Files	Coverage Δ
cmd/tendermint/commands/gen_node_key.go	`0.00% <ø> (ø)`
cmd/tendermint/commands/gen_validator.go	`18.75% <ø> (ø)`
cmd/tendermint/commands/probe_upnp.go	`0.00% <ø> (ø)`
cmd/tendermint/commands/replay.go	`0.00% <ø> (ø)`
cmd/tendermint/commands/reset_priv_validator.go	`8.33% <ø> (ø)`
cmd/tendermint/commands/root.go	`50.00% <ø> (+5.88%)`	⬆️
cmd/tendermint/commands/show_node_id.go	`0.00% <ø> (ø)`
cmd/tendermint/commands/show_validator.go	`0.00% <ø> (ø)`
rpc/core/env.go	`47.36% <ø> (ø)`
cmd/tendermint/commands/inspect.go	`19.14% <19.14%> (ø)`
... and 24 more

lgtm-com · 2021-07-30T00:10:47Z

This pull request introduces 1 alert when merging e2a6522 into 9a2a7d4 - view on LGTM.com

new alerts:

1 for Useless assignment to local variable

rpc/jsonrpc/server/http_server.go

store/store.go

state/mocks/block_store.go

rpc/core/routes.go

node/debug_test.go

node/debug.go

node/rpc.go

tac0turtle · 2021-07-30T09:14:03Z

I like the approach here and the clean up!!

node/debug.go

internal/consensus/replay_test.go

state/services.go

store/store.go

types/event_bus.go

node/debug.go

node/node_test.go

node/rpc.go

rpc/core/routes.go

inspect/inspect_test.go

cmwaters

Great Work!

I think just some minor touch ups / linting is required before this can be merged.

As a high level question, what would happen if I had a node running and then I inspected it at the same time. Would it work as expected or error? As a guess, this would depend on whether the db supports additional read-only connections right?

inspect/inspect.go

inspect/inspect_test.go

cmd/tendermint/commands/inspect.go

williambanfield

Great Work!

I think just some minor touch ups / linting is required before this can be merged.

As a high level question, what would happen if I had a node running and then I inspected it at the same time. Would it work as expected or error? As a guess, this would depend on whether the db supports additional read-only connections right?

Yeah, this would be reliant on how the DB manages the files for the storage. I'm not yet sure users will want to run this at the same time as a node. The node already provides RPC so this would be somewhat redundant. I tried running a node of each DB type alongside the corresponding inspect command one by one.

When trying this right now:

Error that is reported by goleveldb:

./build/tendermint inspect
ERROR: failed to initialize database: resource temporarily unavailable

Error that is reported by cleveldb:

./build/tendermint inspect --db-backend cleveldb
ERROR: failed to initialize database: IO error: lock /home/william/.tendermint/data/blockstore.db/LOCK: Resource temporarily unavailable

badgerdb:

 ./build/tendermint inspect --db-backend badgerdb
ERROR: failed to initialize database: Cannot acquire directory lock on "/home/william/.tendermint/data/blockstore".  Another process is using this Badger database.: resource temporarily unavailable

boltdb hangs trying to initialize the db connection

rocksdb:

./build/tendermint inspect --db-backend rocksdb
ERROR: failed to initialize database: IO error: While lock file: /home/william/.tendermint/data/blockstore.db/LOCK: Resource temporarily unavailable

badgerdb:

./build/tendermint inspect --db-backend badgerdb
ERROR: failed to initialize database: Cannot acquire directory lock on "/home/william/.tendermint/data/blockstore".  Another process is using this Badger database.: resource temporarily unavailable

creachadair · 2021-08-24T18:13:53Z

inspect/inspect_test.go

+		require.NoError(t, d.Run(ctx))
+	}()
+	// FIXME: used to induce context switch.
+	// Determine more deterministic method for prompting a context switch


FYI I also filed #6858 to track this more generally.

EDIT: Updated, see [comment below]( #6785 (comment)) This change adds a sketch of the `Debug` mode. This change adds a `Debug` struct to the node package. This `Debug` struct is intended to be created and started by a command in the `cmd` directory. The `Debug` struct runs the RPC server on the data directories: both the state store and the block store. This change required a good deal of refactoring. Namely, a new `rpc.go` file was added to the `node` package. This file encapsulates functions for starting RPC servers used by nodes. A potential additional change is to further factor this code into shared code _in_ the `rpc` package. Minor API tweaks were also made that seemed appropriate such as the mechanism for fetching routes from the `rpc/core` package. Additional work is required to register the `Debug` service as a command in the `cmd` directory but I am looking for feedback on if this direction seems appropriate before diving much further. closes: #5908

resurrect the inspect command from #6785 Co-authored-by: Sam Kleinman <garen@tychoish.com> Co-authored-by: Thane Thomson <connect@thanethomson.com> Co-authored-by: Callum Waters <cmwaters19@gmail.com>

williambanfield requested review from alexanderbez, cmwaters, ebuchman, tessr and tychoish as code owners July 29, 2021 23:47

williambanfield commented Jul 29, 2021

View reviewed changes

rpc/jsonrpc/server/http_server.go Show resolved Hide resolved

rpc/jsonrpc/server/http_server.go Outdated Show resolved Hide resolved

node/setup.go Outdated Show resolved Hide resolved

rpc/core/routes.go Outdated Show resolved Hide resolved

williambanfield marked this pull request as draft July 29, 2021 23:55

williambanfield commented Jul 29, 2021

View reviewed changes

rpc/core/routes.go Outdated Show resolved Hide resolved

williambanfield requested a review from creachadair July 30, 2021 00:00

cmwaters reviewed Jul 30, 2021

View reviewed changes

rpc/jsonrpc/server/http_server.go Show resolved Hide resolved

store/store.go Outdated Show resolved Hide resolved

cmwaters reviewed Jul 30, 2021

View reviewed changes

state/mocks/block_store.go Outdated Show resolved Hide resolved

rpc/core/routes.go Outdated Show resolved Hide resolved

node/debug_test.go Outdated Show resolved Hide resolved

node/debug.go Outdated Show resolved Hide resolved

tac0turtle reviewed Jul 30, 2021

View reviewed changes

node/rpc.go Outdated Show resolved Hide resolved

tychoish reviewed Jul 30, 2021

View reviewed changes

node/debug.go Outdated Show resolved Hide resolved

node/debug.go Outdated Show resolved Hide resolved

creachadair reviewed Jul 30, 2021

View reviewed changes

williambanfield added 2 commits August 2, 2021 12:08

node: sketch of debug node logic

df669c7

PR feedback fixes

03ee71e

williambanfield force-pushed the wb/issue-5908 branch from fbaf7d4 to 03ee71e Compare August 2, 2021 16:50

lint++

2be1b37

creachadair reviewed Aug 2, 2021

View reviewed changes

rpc/core/routes.go Outdated Show resolved Hide resolved

rpc/core/routes.go Outdated Show resolved Hide resolved

williambanfield and others added 8 commits August 2, 2021 17:06

add tx indexer routes

1f7ecbd

Merge branch 'master' into wb/issue-5908

c3561f1

refactor debug code into separate package and rename to inspect

c37e4a4

wip

c607707

update comments for the inspect server

557326c

remove node refactor

45f92ee

remove env refactor

75f549d

remove env refactor

6c7dbec

williambanfield and others added 3 commits August 23, 2021 10:29

Merge branch 'master' into wb/issue-5908

9242e15

Merge branch 'master' into wb/issue-5908

d0a5d0f

add tests

4c381f1

creachadair reviewed Aug 23, 2021

View reviewed changes

inspect/inspect_test.go Outdated Show resolved Hide resolved

cmwaters approved these changes Aug 24, 2021

View reviewed changes

inspect/inspect.go Outdated Show resolved Hide resolved

inspect/inspect_test.go Outdated Show resolved Hide resolved

cmd/tendermint/commands/inspect.go Outdated Show resolved Hide resolved

lint fixes

4a97aaf

williambanfield commented Aug 24, 2021

View reviewed changes

williambanfield and others added 2 commits August 24, 2021 11:22

inspect -> inspector

8cbe793

Merge branch 'master' into wb/issue-5908

ab1739b

williambanfield added the S:automerge Automatically merge PR when requirements pass label Aug 24, 2021

williambanfield changed the title ~~node: add inspect mode for debugging crashed tendermint node~~ inspect: add inspect mode for debugging crashed tendermint node Aug 24, 2021

williambanfield added 2 commits August 24, 2021 11:46

increase timeouts for context switch

61ed813

Merge remote-tracking branch 'origin/wb/issue-5908' into wb/issue-5908

baed887

williambanfield mentioned this pull request Aug 24, 2021

rpc/core routes composability #6855

Closed

williambanfield and others added 7 commits August 24, 2021 11:57

inspect test timeout in ci/cd

528d41f

Merge branch 'master' into wb/issue-5908

41da431

add small timeout to requireConnect

d55d801

lint fix

e956b95

Merge branch 'master' into wb/issue-5908

234af06

increase timeouts for context switch

243b4ac

Merge remote-tracking branch 'origin/wb/issue-5908' into wb/issue-5908

3a858ca

mergify bot merged commit bc2b529 into master Aug 24, 2021

mergify bot deleted the wb/issue-5908 branch August 24, 2021 18:12

creachadair reviewed Aug 24, 2021

View reviewed changes

williambanfield mentioned this pull request Oct 1, 2021

cli: allow node operator to rollback last state #7033

Merged

tac0turtle mentioned this pull request Jul 27, 2022

v0.37 release plan #9091

Closed

39 tasks

williambanfield restored the wb/issue-5908 branch September 9, 2022 16:13

williambanfield mentioned this pull request Nov 1, 2022

inspect: resurrect the inspect command #9655

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

inspect: add inspect mode for debugging crashed tendermint node #6785

inspect: add inspect mode for debugging crashed tendermint node #6785

williambanfield commented Jul 29, 2021 •

edited

codecov bot commented Jul 30, 2021 •

edited

lgtm-com bot commented Jul 30, 2021

tac0turtle commented Jul 30, 2021

cmwaters left a comment

williambanfield left a comment •

edited

creachadair Aug 24, 2021

inspect: add inspect mode for debugging crashed tendermint node #6785

inspect: add inspect mode for debugging crashed tendermint node #6785

Conversation

williambanfield commented Jul 29, 2021 • edited

codecov bot commented Jul 30, 2021 • edited

Codecov Report

lgtm-com bot commented Jul 30, 2021

tac0turtle commented Jul 30, 2021

cmwaters left a comment

Choose a reason for hiding this comment

williambanfield left a comment • edited

Choose a reason for hiding this comment

creachadair Aug 24, 2021

Choose a reason for hiding this comment

williambanfield commented Jul 29, 2021 •

edited

codecov bot commented Jul 30, 2021 •

edited

williambanfield left a comment •

edited