gaia stuck in error: couldn't find validators at height 15214022

[flynnji]

Problem

firstly, i met the same error this issue #2406 , so i try to do something to fix the node. and then i execute the commands as below

# gaiad rollback 15214023
# gaiad rollback 15214022

after that i got a panic error while i trying to run gaiad.
what can i do to fix it?

:16AM INF starting node with ABCI Tendermint in-process
9:16AM INF service start impl=multiAppConn module=proxy msg={}
9:16AM INF service start connection=query impl=localClient module=abci-client msg={}
9:16AM INF service start connection=snapshot impl=localClient module=abci-client msg={}
9:16AM INF service start connection=mempool impl=localClient module=abci-client msg={}
9:16AM INF service start connection=consensus impl=localClient module=abci-client msg={}
9:16AM INF service start impl=EventBus module=events msg={}
9:16AM INF service start impl=PubSub module=pubsub msg={}
9:19AM INF service start impl=IndexerService module=txindex msg={}
9:19AM INF ABCI Handshake App Info hash="^lo��D`�n\x1d#��a\x1d+��\ns�v\x03濮�B\v��1" height=15214023 module=consensus protocol-version=0 software-version=HEAD-321d15a574def0f338ceacc5c060159ebba95edc
9:19AM INF ABCI Replay Blocks appHeight=15214023 module=consensus stateHeight=15214023 storeHeight=15214024
9:19AM INF Replay last block using real app module=consensus
panic: couldn't find validators at height 15214022 (height 15214023 was originally requested): %!w(<nil>)

goroutine 1 [running]:
github.com/tendermint/tendermint/state.getBeginBlockValidatorInfo(0xc000001680, {0x2689528, 0xc009424648}, 0x4f5b97)
	github.com/tendermint/tendermint@v0.34.27/state/execution.go:346 +0x3ea
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x267c260?, 0xc24b6155c0}, {0x26823f0, 0xc010c83d90}, 0xc000001680, {0x2689528, 0xc009424648}, 0xe825c7?)
	github.com/tendermint/tendermint@v0.34.27/state/execution.go:293 +0x219
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0227e99b0, 0x7}}, {0xc0227e99c0, 0xb}, 0x4f5b97, 0xe825c7, {{0xc019214240, ...}, ...}, ...}, ...)
	github.com/tendermint/tendermint@v0.34.27/state/execution.go:140 +0x171
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc0227e99b0, 0x7}}, {0xc0227e99c0, 0xb}, 0x4f5b97, 0xe825c7, {{0xc019214240, ...}, ...}, ...}, ...)
	github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:503 +0x23c
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc0227e99b0, 0x7}}, {0xc0227e99c0, 0xb}, 0x4f5b97, 0xe825c7, {{0xc019214240, ...}, ...}, ...}, ...)
	github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:416 +0x7ae
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc001227e70, {0x268a760, 0xc0105f60d0})
	github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:268 +0x3c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc0227e99b0, 0x7}}, {0xc0227e99c0, 0xb}, 0x4f5b97, 0xe825c7, ...}, ...)
	github.com/tendermint/tendermint@v0.34.27/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc000c22640, {0x2677ee0, 0xc00019ca00}, 0xc0015ea3f0, {0x2661a60, 0xc00d2ee390}, 0x0?, 0x0?, 0xc0015ea660, {0x267c260, ...}, ...)
	github.com/tendermint/tendermint@v0.34.27/node/node.go:779 +0x597
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2691928, 0xc000c13440}, {0x0, 0x0}, {0x267fd30, 0xc0010fede0}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.45.15-ics/server/start.go:280 +0x81b
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc001142f00?, {0x356bf20?, 0x0?, 0x0?})
	github.com/cosmos/cosmos-sdk@v0.45.15-ics/server/start.go:128 +0x169
github.com/spf13/cobra.(*Command).execute(0xc001142f00, {0x356bf20, 0x0, 0x0})
	github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000ec5200)
	github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.6.1/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/cobra@v1.6.1/command.go:961
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc00110c210, 0xb})
	github.com/cosmos/cosmos-sdk@v0.45.15-ics/server/cmd/execute.go:36 +0x1eb
main.main()
	github.com/cosmos/gaia/v9/cmd/gaiad/main.go:16 +0x2c

[bb4L]

same here

[mmulji-ic]

@flynnji @bb4L think it probably means that block isn't available with the validators that your connected to. Will check with the Comet team, cc @alijnmerchant21

panic: couldn't find validators at height 15214022 (height 15214023 was originally requested): %!w(<nil>)

[mmulji-ic]

@flynnji @bb4L sorry picking this up from awhile back, we discussed the rollback feature and it doesn't work as advertised. As the error message stated you don't seem to be connected to a node that has that block, maybe due to maxxed out connections. Pinging @alijnmerchant21 who's on the Comet team.

[mmulji-ic]

Hi @andynog can you comment on the rollback functionality?

[MSalopek]

Closing as there has been no activity for > 6months from the original issue authors.

The discussion has decayed.