Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default RocksDB WAL log size settings #837

Open
iadmytro opened this issue Mar 27, 2024 · 0 comments
Open

Default RocksDB WAL log size settings #837

iadmytro opened this issue Mar 27, 2024 · 0 comments

Comments

@iadmytro
Copy link

With current settings rocksdb produces a lot of WAL log files that are not removed in timely manner. It leads to substantial memory consumption.
Behavior governed by two settings set_max_total_wal_size and set_wal_ttl_seconds

  1. If both set to 0, logs will be deleted asap and will not get into the archive.
  2. If wal_ttl_seconds is 0 and wal_size_limit_mb is not 0, WAL files will be checked every 10 min and if total size is greater then wal_size_limit_mb, they will be deleted starting with the earliest until size_limit is met. All empty files will be deleted.
  3. If wal_ttl_seconds is not 0 and wall_size_limit_mb is 0, then WAL files will be checked every wal_ttl_seconds / 2 and those that are older than wal_ttl_seconds will be deleted.
  4. If both are not 0, WAL files will be checked every 10 min and both checks will be performed with ttl being first.

Source

By default it's option 1

set_max_total_wal_size(0)
set_wal_ttl_seconds(0)

Suggestion.
Switch to option 4

set_max_total_wal_size(some_value)
set_wal_ttl_seconds(some_value)

which gives more predictable cleanup, it will happen according to set_wal_ttl_seconds value, and potentially smaller memory consumption, garbage will not accumulate.

@iadmytro iadmytro changed the title Default WAL log size settings Default RocksDB WAL log size settings Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant