Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graph.build with an empty Backend will rebuild and overwrite data #355

Open
JacobHayes opened this issue Mar 28, 2023 · 0 comments
Open
Assignees
Labels
bug Something isn't working

Comments

@JacobHayes
Copy link
Member

Describe the bug

Graph.build() with existing outputs not tracked in the Backend (eg: run build, then wipe db) will happily overwrite data because it thinks it doesn't exist. We should check for this and error during build.

This doesn't happen with the MemoryBackend because that calls storage.discover_partitions(...) if there is no hit in the db (since it's not actually persistent).

Does this negate the perf benefits of the db? Probably not for most sync style operations, but may lead to rate limits issues depending on how it is implemented.

To Reproduce

Update docs/examples/spend/demo.py to use a persistent backend, run it once, wipe the backend, and run again. Notice that the subsequent run did not skip.

Expected behavior

The second run to error saying something about "Unexpectedly found data at ..." and how to remedy (either delete data or some way to import into the db).

@JacobHayes JacobHayes added the bug Something isn't working label Mar 28, 2023
@JacobHayes JacobHayes self-assigned this Aug 7, 2023
@JacobHayes JacobHayes changed the title Graph.build with a wiped Backend will overwrite data Graph.build with an empty Backend will rebuild and overwrite data Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant