Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUESTION]: Batch write is partially committed when badger crashes #2024

Open
dannybabbev opened this issue Oct 22, 2023 · 0 comments
Open
Labels
kind/question Something requiring a response

Comments

@dannybabbev
Copy link

dannybabbev commented Oct 22, 2023

Question.

I am using badger for a data-intensive process, writing and deleting up the 10 million values per write batch in a database of ~200 million key/value pairs.
I have the following code that is running a write batch:

func (b *Indexer) updateDB() {
	wb := b.db.NewWriteBatch()
	defer wb.Cancel()

	...set and delete up to 10 million values

	err = wb.Flush()
	if err != nil {
		log.Panicf("Error flushing writes to db %v", err)
	}
}

This code runs for several minutes and most of the time everything is successful.

However, I noticed that when badger crashes, sometimes due to exceeded memory, or when the db is not closed correctly the data in the write batch is partially committed. Only some of the reads and writes make it to the database.

I would have expected that the write batch either succeeds in full or fails altogether, in an ACID style. Why is this not the case here?

Is there a way to have an ACID batch write or to recover the database, if it was closed ungracefully, to a previous consistent state?

@dannybabbev dannybabbev added the kind/question Something requiring a response label Oct 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/question Something requiring a response
Development

No branches or pull requests

1 participant