Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cosmovisor.service: Main process exited, code=exited, status=1/FAILURE #2928

Open
Joshua1-Dev opened this issue Feb 5, 2024 · 12 comments
Open

cosmovisor.service: Main process exited, code=exited, status=1/FAILURE #2928

Joshua1-Dev opened this issue Feb 5, 2024 · 12 comments

Comments

@Joshua1-Dev
Copy link

my node keeps crashing after a while with this errors

Feb 05 12:10:44 interview-test3.us-central1-a.c.validator-379314.internal systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE
Feb 05 12:10:44 interview-test3.us-central1-a.c.validator-379314.internal systemd[1]: cosmovisor.service: Failed with result 'exit-code'.

this is my cosmovisor.service

[Unit]
Description=Cosmovisor service
After=network-online.target

[Service]
User=root
ExecStart=/home/ubuntu/go/bin/cosmovisor run start --x-crisis-skip-assert-invariants --home /home/ubuntu/.gaia
Restart=on-failure
RestartSec=5s
StartLimitBurst=5
StartLimitInterval=10m
LimitNOFILE=4096
Environment='DAEMON_NAME=gaiad'
Environment='DAEMON_HOME=/home/ubuntu/.gaia'
Environment='DAEMON_ALLOW_DOWNLOAD_BINARIES=true'
Environment='DAEMON_RESTART_AFTER_UPGRADE=true'
Environment='DAEMON_LOG_BUFFER_SIZE=512'
Environment='UNSAFE_SKIP_BACKUP=true'

[Install]
WantedBy=multi-user.target

error

what could be the issue?
@github-actions github-actions bot added the status: waiting-triage This issue/PR has not yet been triaged by the team. label Feb 5, 2024
@julienrbrt
Copy link
Member

Hi, this issue doesn't look like related to cosmovisor, as your node crashes, not cosmovisor.
I think you should troubleshoot your issue in the Cosmos Hub repo: https://github.com/cosmos/gaia

@julienrbrt julienrbrt removed the status: waiting-triage This issue/PR has not yet been triaged by the team. label Feb 5, 2024
@Joshua1-Dev
Copy link
Author

Hi, this issue doesn't look like related to cosmovisor, as your node crashes, not cosmovisor. I think you should troubleshoot your issue in the Cosmos Hub repo: https://github.com/cosmos/gaia
to crash
the cosmovisor.service crashes before the node crash. basically the cosmovisor is causing it

@julienrbrt
Copy link
Member

Can you post the full logs?

@Joshua1-Dev
Copy link
Author

Joshua1-Dev commented Feb 5, 2024 •
edited by julienrbrt

Can you post the full logs?

n
n

Feb 05 13:03:35 interview-test3.us-central1-a.c.validator-379314.internal systemd[1]: Started cosmovisor.service - Cosmovisor service.
Feb 05 13:03:35 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840810]: 1:03PM INF running app args=["start","--x-crisis-skip-assert-invariants","--home","/home/ubuntu/.gaia"] module=cosmovisor path=/home/ubuntu/.gaia/cosmovisor/genesis/bin/gaiad
Feb 05 13:03:35 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF starting node with ABCI Tendermint in-process
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF service start impl=multiAppConn module=proxy msg={}
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF service start connection=query impl=localClient module=abci-client msg={}
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF service start connection=snapshot impl=localClient module=abci-client msg={}
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF service start connection=mempool impl=localClient module=abci-client msg={}
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF service start connection=consensus impl=localClient module=abci-client msg={}
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF service start impl=EventBus module=events msg={}
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF service start impl=PubSub module=pubsub msg={}
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF service start impl=IndexerService module=txindex msg={}
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF Found local state with non-zero height, skipping state sync
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF ABCI Handshake App Info hash="���?\n�3E5����ڦ��n�\x02�(�8g�M���v�" height=20142000 module=consensus protocol-version=0 software-version=v13.0.0
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF ABCI Replay Blocks appHeight=20142000 module=consensus stateHeight=20142000 storeHeight=20142001
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: 1:03PM INF Replay last block using real app module=consensus
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: panic: runtime error: invalid memory address or nil pointer dereference
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1838ed4]
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: goroutine 1 [running]:
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0x7ffe8c908e2a, 0x12}, 0xc000d06570, {0x25cd140, 0xc000d2fd80}, {0x25ef510, 0xc000eed190}, 0xc000d07da0, {0x25c70a0, 0xc00007ee00}, ...}, ...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/cosmos/cosmos-sdk@v0.45.16-ics/x/upgrade/abci.go:38 +0x1014
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/cosmos/cosmos-sdk@v0.45.16-ics/x/upgrade/module.go:130
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x25e1698, 0xc0001b0000}, {0x25ef0f0, 0xc000bf2340}, {{0xb, 0x0}, {0xc000d81b18, 0x11}, 0x13357b1, ...}, ...}, ...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/cosmos/cosmos-sdk@v0.45.16-ics/types/module/module.go:491 +0x1db
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/cosmos/gaia/v13/app.(*GaiaApp).BeginBlocker(...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/cosmos/gaia/v13/app/app.go:235
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc000be4100, 0x20, 0x20}, {{0xb, 0x0}, {0xc000d81b18, 0x11}, 0x13357b1, {0x2c5b33dd, ...}, ...}, ...})
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/cosmos/cosmos-sdk@v0.45.16-ics/baseapp/abci.go:177 +0x97b
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc000be4100, 0x20, 0x20}, {{0xb, 0x0}, {0xc000d81b18, 0x11}, 0x13357b1, {0x2c5b33dd, ...}, ...}, ...})
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/tendermint/tendermint@v0.34.27/abci/client/local_client.go:280 +0x118
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc000be4100, 0x20, 0x20}, {{0xb, 0x0}, {0xc000d81b18, 0x11}, 0x13357b1, {0x2c5b33dd, ...}, ...}, ...})
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/tendermint/tendermint@v0.34.27/proxy/app_conn.go:81 +0x55
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x25e2460?, 0xc000d7f3e0}, {0x25e9700, 0xc0001932e0}, 0xc000229a40, {0x25f0af8, 0xc000d6e498}, 0x13357b0?)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/tendermint/tendermint@v0.34.27/state/execution.go:307 +0x51d
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc000057380, 0x7}}, {0xc000d80e40, 0x11}, 0x89dbae, 0x13357b0, {{0xc000eeeb60, ...}, ...}, ...}, ...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/tendermint/tendermint@v0.34.27/state/execution.go:140 +0x171
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc000057380, 0x7}}, {0xc000d80e40, 0x11}, 0x89dbae, 0x13357b0, {{0xc000eeeb60, ...}, ...}, ...}, ...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:503 +0x23c
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc000057380, 0x7}}, {0xc000d80e40, 0x11}, 0x89dbae, 0x13357b0, {{0xc000eeeb60, ...}, ...}, ...}, ...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:416 +0x7ae
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc000cd5c78, {0x25f1f40, 0xc0000b4000})
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/tendermint/tendermint@v0.34.27/consensus/replay.go:268 +0x3d4
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc000057380, 0x7}}, {0xc000d80e40, 0x11}, 0x89dbae, 0x13357b0, ...}, ...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/tendermint/tendermint@v0.34.27/node/node.go:329 +0x1b8
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/tendermint/tendermint/node.NewNode(0xc000ab2a00, {0x25de380, 0xc000bab720}, 0xc000d62d50, {0x25c8280, 0xc000d6fbf0}, 0x0?, 0x0?, 0xc000d62f70, {0x25e2460, ...}, ...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/tendermint/tendermint@v0.34.27/node/node.go:779 +0x597
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x25fe700, 0xc000aa4a50}, {0x0, 0x0}, {0x25e69b8, 0xc000eed190}, ...}, ...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/cosmos/cosmos-sdk@v0.45.16-ics/server/start.go:280 +0x89b
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000f4a600?, {0xc000aa4240?, 0x0?, 0x3?})
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/cosmos/cosmos-sdk@v0.45.16-ics/server/start.go:128 +0x169
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/spf13/cobra.(*Command).execute(0xc000f4a600, {0xc000aa41b0, 0x3, 0x3})
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/spf13/cobra@v1.7.0/command.go:940 +0x862
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000224f00)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/spf13/cobra.(*Command).Execute(...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/spf13/cobra@v1.7.0/command.go:992
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/spf13/cobra@v1.7.0/command.go:985
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc000eea730, 0xb})
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/cosmos/cosmos-sdk@v0.45.16-ics/server/cmd/execute.go:36 +0x1eb
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]: main.main()
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840818]:         github.com/cosmos/gaia/v13/cmd/gaiad/main.go:16 +0x2c
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840810]: Error: exit status 2
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal cosmovisor[840810]: 1:03PM ERR error="exit status 2" module=cosmovisor
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal systemd[1]: cosmovisor.service: Failed with result 'exit-code'.
Feb 05 13:03:42 interview-test3.us-central1-a.c.validator-379314.internal systemd[1]: cosmovisor.service: Consumed 7.956s CPU time.
Feb 05 13:03:47 interview-test3.us-central1-a.c.validator-379314.internal systemd[1]: cosmovisor.service: Scheduled restart job, restart counter is at 3.
Feb 05 13:03:47 interview-test3.us-central1-a.c.validator-379314.internal systemd[1]: Stopped cosmovisor.service - Cosmovisor service.
Feb 05 13:03:47 interview-test3.us-central1-a.c.validator-379314.internal systemd[1]: cosmovisor.service: Consumed 7.956s CPU time.

@julienrbrt
Copy link
Member

What do you get if you run your node without cosmovisor?
Namely, gaiad start --x-crisis-skip-assert-invariants --home /home/ubuntu/.gaia

It still looks like a node error to me and not an issue with cosmovisor.

@Joshua1-Dev
Copy link
Author

What do you get if you run your node without cosmovisor? Namely, gaiad start --x-crisis-skip-assert-invariants --home /home/ubuntu/.gaia

It still looks like a node error to me and not an issue with cosmovisor.

gaiad start --x-crisis-skip-assert-invariants --home /home/ubuntu/.gaia
Error: failed to initialize database: open /home/ubuntu/.gaia/data/application.db/LOCK: permission denied

@julienrbrt
Copy link
Member

What do you get if you run your node without cosmovisor? Namely, gaiad start --x-crisis-skip-assert-invariants --home /home/ubuntu/.gaia

It still looks like a node error to me and not an issue with cosmovisor.

gaiad start --x-crisis-skip-assert-invariants --home /home/ubuntu/.gaia

Error: failed to initialize database: open /home/ubuntu/.gaia/data/application.db/LOCK: permission denied

Make sure to run with the same user as the systemd service user, or use chown -R for the .gaia folder.

@Joshua1-Dev
Copy link
Author

What do you get if you run your node without cosmovisor? Namely, gaiad start --x-crisis-skip-assert-invariants --home /home/ubuntu/.gaia

It still looks like a node error to me and not an issue with cosmovisor.

it"s running on master

@umegbewe
Copy link

umegbewe commented Feb 7, 2024

I'm experiencing this same problem @julienrbrt

Result of running

gaiad start --x-crisis-skip-assert-invariants --home /home/ubuntu/.gaia

I get the same error when running gaiad with cosmovisor

7:41PM INF starting node with ABCI Tendermint in-process
7:41PM INF service start impl=multiAppConn module=proxy msg={}
7:41PM INF service start connection=query impl=localClient module=abci-client msg={}
7:41PM INF service start connection=snapshot impl=localClient module=abci-client msg={}
7:41PM INF service start connection=mempool impl=localClient module=abci-client msg={}
7:41PM INF service start connection=consensus impl=localClient module=abci-client msg={}
7:41PM INF service start impl=EventBus module=events msg={}
7:41PM INF service start impl=PubSub module=pubsub msg={}
7:41PM INF service start impl=IndexerService module=txindex msg={}
7:41PM INF Found local state with non-zero height, skipping state sync
7:41PM INF ABCI Handshake App Info hash="���?\n�3E5����ڦ��n�\x02�(�8g�M���v�" height=20142000 module=consensus protocol-version=0 software-version=v13.0.0
7:41PM INF ABCI Replay Blocks appHeight=20142000 module=consensus stateHeight=20142000 storeHeight=20142001
7:41PM INF Replay last block using real app module=consensus
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1838ed4]

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0x7ffec89036af, 0x12}, 0xc000dbe780, {0x25cd140, 0xc000df6b70}, {0x25ef510, 0xc000d7c460}, 0xc000dbffb0, {0x25c70a0, 0xc0000aee00}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.45.16-ics/x/upgrade/abci.go:38 +0x1014
github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)
	github.com/cosmos/cosmos-sdk@v0.45.16-ics/x/upgrade/module.go:130

@julienrbrt
Copy link
Member

Hi, this issue isn't cosmovisor related. Cosmovisor starts the app and then the app panics. Please refer to the gaia team.

@tac0turtle tac0turtle transferred this issue from cosmos/cosmos-sdk Feb 7, 2024
@mmulji-ic
Copy link
Contributor

@umegbewe @Joshua1-Dev which Gaia version are you using and also which network are you pointing Gaia to? Can you also let us know the cosmovisor directory layout.

@MSalopek
Copy link
Contributor

MSalopek commented Mar 25, 2024 •
edited

Are you trying to catch up to the theta testnet?

You needed to use v14.2.0 if you were catching up to the network tip - your logs state you were using v13 which is really old.

To join at present time, you need to use v15.1.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Cosmos Hub
Status: 🩹 F1: Triage
Cosmos-SDK
Status: 📋 Backlog
Milestone
No milestone
Development

No branches or pull requests
5 participants
@julienrbrt @MSalopek @umegbewe @mmulji-ic @Joshua1-Dev