Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

badger backup system #776

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

badger backup system #776

wants to merge 8 commits into from

Conversation

requilence
Copy link
Contributor

@requilence requilence commented Jan 8, 2024

for spacestore

  • run incremental backup every minute
  • every 1000 backups(empty skipped) do the full backup and remove all others after
  • do the os.Sync before renaming backups from *.tmp
  • in case it was not able to open badger on start recover from backup
  • only in case backup restore was successful the old corrupted dir got renamed and the restored one take it place
  • because we never rewrite most of the keys' values in spacestore(only object heads), incremental backup works effective and doesn't consume a lot of space

for localstore

  • in case of error on open just reinit the db without backup
  • corrupted repo left for debugging

Also
Need to switch to this PR's branch dgraph-io/badger#2033
The reason is that in the most cases the panic happens in the goroutine and it not possible to catch it in from the badger.Open caller

I think backuper may be used only on WIndows. Let's see how often this corruptions happen on UNIX

Copy link

github-actions bot commented Jan 11, 2024

New Coverage 43.7% of statements
Patch Coverage 47.1% of changed statements (105/223)

Coverage provided by https://github.com/seriousben/go-patch-cover-action

call os memtable sync syscall after N seconds of inactivity on db
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants