Celestia light node storage filling up quickly

[Alper2022]

Hello!

I am currently running 4 celestia light nodes on a cluster of Raspberry Pi's i had lying around, all of them pi 4's with a 64gb storage card and 8gb of ram.

The issue i currently am noticing is, when running these nodes they QUICKLY fill up all the space on the disk, most of the space being lost to ~/.celestia-light/data/

My question basically is:

Is it normal for ~/.celestia-light/data/ to completely fill up, and end up shutting down the light node? or am i forgetting a configuration?

Thank you for your time and attention!

with kind regards,
Alper

edit: the command i run the nodes with is "nohup celestia light &"

[jcstein]

@ramin can you please transfer this to celestia-node?

[ramin]

hi @Alper2022 no that doesn't sound normal. Are there any logs you can share? Also, not sure if a typo or not, but you should start the light node with celestia light start (having completed init previously). Can you verify that is the starting command, I'd recommend revisiting the docs here https://docs.celestia.org/nodes/light-node

[Alper2022]

I could share the "nohup" output file, but unfortunately there doesn't seem to be anything concrete on there that i could figure the issue out from.

currently this is what the "~/.celestia-light/data/" directory looks like:

this is also the error nohup gives when the storage gets too full(last successful header/store line added for clarity):

2024-04-27T15:17:37.530+0200    INFO    header/store    store/store.go:365      new head        {"height": 1320463, "hash": "227E0FD1E1B07F7DA5B7BADFF343437DA4AD56107706CD8F14CEE8AB804EE0B8"}
unexpected fault address 0xffff18172000
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0xffff18172000 pc=0x4755b8]

goroutine 383469558 [running]:
runtime.throw({0x27a0e25?, 0x4009aeab10?})
        /usr/local/go/src/runtime/panic.go:1077 +0x40 fp=0x401afeac80 sp=0x401afeac50 pc=0x43d060
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:858 +0xec fp=0x401afeace0 sp=0x401afeac80 pc=0x45513c
runtime.memmove()
        /usr/local/go/src/runtime/memmove_arm64.s:160 +0x168 fp=0x401afeacf0 sp=0x401afeacf0 pc=0x4755b8
github.com/dgraph-io/badger/v4/table.(*buildData).Copy(0x401afeadf8, {0xffff18172000, 0x5cd69, 0x5cd69})
        /home/pi/go/pkg/mod/github.com/celestiaorg/badger/v4@v4.0.0-20231125230536-2b9e13346f75/table/builder.go:419 +0xb4 fp=0x401afead50 sp=0x401afeacf0 pc=0x198afb4
github.com/dgraph-io/badger/v4/table.CreateTable({0x4009aeab10, 0x28}, 0x4013548630)
        /home/pi/go/pkg/mod/github.com/celestiaorg/badger/v4@v4.0.0-20231125230536-2b9e13346f75/table/table.go:269 +0x1a4 fp=0x401afeaf10 sp=0x401afead50 pc=0x198f9a4
github.com/dgraph-io/badger/v4.(*levelsController).subcompact.func4(0x4013548630, 0x2?)
        /home/pi/go/pkg/mod/github.com/celestiaorg/badger/v4@v4.0.0-20231125230536-2b9e13346f75/levels.go:859 +0xdc fp=0x401afeafb0 sp=0x401afeaf10 pc=0x19b673c
github.com/dgraph-io/badger/v4.(*levelsController).subcompact.func5()
        /home/pi/go/pkg/mod/github.com/celestiaorg/badger/v4@v4.0.0-20231125230536-2b9e13346f75/levels.go:867 +0x38 fp=0x401afeafd0 sp=0x401afeafb0 pc=0x19b6628
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm64.s:1197 +0x4 fp=0x401afeafd0 sp=0x401afeafd0 pc=0x474ba4
created by github.com/dgraph-io/badger/v4.(*levelsController).subcompact in goroutine 383419643
        /home/pi/go/pkg/mod/github.com/celestiaorg/badger/v4@v4.0.0-20231125230536-2b9e13346f75/levels.go:849 +0x578

goroutine 1 [chan receive, 2373 minutes]:
runtime.gopark(0x34d6850?, 0x221e8a0?, 0xf0?, 0xd1?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xc8 fp=0x4006993770 sp=0x4006993750 pc=0x43fe78
runtime.chanrecv(0x4001289d40, 0x0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x414 fp=0x40069937f0 sp=0x4006993770 pc=0x40af84
runtime.chanrecv1(0x4001601a40?, 0x350b470?)
        /usr/local/go/src/runtime/chan.go:442 +0x14 fp=0x4006993820 sp=0x40069937f0 pc=0x40ab34
github.com/celestiaorg/celestia-node/cmd.Start.func1(0x40004d5800, {0x51466c0?, 0x4?, 0x279e874?})
        /home/pi/celestia-node/cmd/start.go:66 +0x430 fp=0x4006993c60 sp=0x4006993820 pc=0x2034670
github.com/spf13/cobra.(*Command).execute(0x40004d5800, {0x51466c0, 0x0, 0x0})
        /home/pi/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0x82c fp=0x4006993e10 sp=0x4006993c60 pc=0x5e084c
github.com/spf13/cobra.(*Command).ExecuteC(0x4b6c5a0)
        /home/pi/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344 fp=0x4006993ef0 sp=0x4006993e10 pc=0x5e0fc4
github.com/spf13/cobra.(*Command).Execute(...)
        /home/pi/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/pi/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1032
main.run()
        /home/pi/celestia-node/cmd/celestia/main.go:47 +0x58 fp=0x4006993f10 sp=0x4006993ef0 pc=0x20424e8
main.main()
        /home/pi/celestia-node/cmd/celestia/main.go:40 +0x1c fp=0x4006993f30 sp=0x4006993f10 pc=0x204246c
runtime.main()
        /usr/local/go/src/runtime/proc.go:267 +0x2bc fp=0x4006993fd0 sp=0x4006993f30 pc=0x43

some system specs:

/etc/os-release:

PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

uname -a:
Linux CNode2 5.15.0-1050-raspi #53-Ubuntu SMP PREEMPT Thu Mar 21 10:02:47 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Please let me know if i can be of more help, or if there is any specific tests you'd want me to run, im not that active on github so i don't immediately get a notification when a comment gets added

[walldiss]

Hello @Alper2022! Thank you for bringing this to our attention. It seems that your node was initialized with an earlier version of our software, which could use up to 64GB of storage because it lacked the sampling window feature. This feature allows the node to store data starting from 30 days prior to initialization. Currently, there is no automatic cleanup for storage, so the size will continue to grow over time. We are actively developing a solution known as "light node pruning."

To address this issue immediately, please upgrade to our latest version, v0.13.4, and reinitialize your node. A node initialized with this version should only occupy around 33GB, storing samples from the last 30 days only. Here's how you can upgrade and reinitialize your node:

• Delete the ~/.celestia-light directory to force a resync.
• Run celestia-node init followed by celestia-node start.

This process will reset your keys, so please contact us if you need assistance in preserving them. We are also developing a pruning features to further address storage growth in light nodes, which will be available in a future updates.

[jcstein]

@Alper2022 what version of celestia-node were you on when you made this ticket?

[Alper2022]

The version i was running was the one in february, i had updated by pulling the most recent main branch in april, but i hadn't forced a resync by deleting the ~/.celestia-light directory.

I assume if i backup the ~/.celestia-light/keys directory, force a resync, and replace the new keys with the old ones that it should work perfectly fine?

Thank you @walldiss @jcstein for the rapid and detailed responses!

[walldiss]

You are right, you just need to persist ~/.celestia-light/keys folder to keep the keys

[Alper2022]

Gonna run a test today on 1 node, i'll let you know tomorrow how it went ^^

[ramin]

@Alper2022 how did test go?

[Alper2022]

After running it for 4 days the celestia-light node shutdown again due to the data directory being filled up again, so unfortunately it didn't fix it

[jcstein]

FYI @Alper2022 we've increased the requirements for light nodes in our documentation in celestiaorg/docs#1564

[Alper2022]

No worries, i could also just do a hack and auto-clean the data directory with a cronjob, unless that breaks something?