Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure save_dataframe writes atomically by default #182

Merged
merged 20 commits into from
Nov 15, 2023
Merged

Conversation

jbweston
Copy link
Collaborator

This PR makes it so that 'save_dataframe' writes atomically by default.
This is important for maintaining data integrity during an adaptive scheduler run.

This adds a dependency on the 'atomicwrites' package.

Closes #181.

@codecov
Copy link

codecov bot commented Aug 14, 2023

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (cd3e66c) 72.88% compared to head (97f86a1) 73.46%.

❗ Current head 97f86a1 differs from pull request most recent head db3fdec. Consider uploading reports for the commit db3fdec to get more accurate results

Files Patch % Lines
adaptive_scheduler/utils.py 74.28% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #182      +/-   ##
==========================================
+ Coverage   72.88%   73.46%   +0.58%     
==========================================
  Files          22       22              
  Lines        2482     2480       -2     
==========================================
+ Hits         1809     1822      +13     
+ Misses        673      658      -15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

adaptive_scheduler/utils.py Fixed Show fixed Hide fixed
tests/test_utils.py Dismissed Show dismissed Hide dismissed
@basnijholt
Copy link
Owner

There was some issue with atomicwrites being yanked from pypi by its maintainer.

@basnijholt
Copy link
Owner

Can also be done like this: python-adaptive/adaptive#353

@jbweston
Copy link
Collaborator Author

Can also be done like this: python-adaptive/adaptive#353

Does that handle calling sync and everything?

@basnijholt
Copy link
Owner

Not sure about all the details. I just remembered the panic a few years back when the package was yanked, hehe.

And python-adaptive/adaptive#352

but adding it here again is fine

adaptive_scheduler/utils.py Fixed Show fixed Hide fixed
adaptive_scheduler/utils.py Fixed Show fixed Hide fixed
@basnijholt
Copy link
Owner

Python 3.9 CI job seems to consistently get stuck?

@jbweston
Copy link
Collaborator Author

Python 3.9 CI job seems to consistently get stuck?

Yes, no idea what is causing this, and cannot reproduce locally.

@jbweston jbweston merged commit 2d7d4ae into main Nov 15, 2023
10 checks passed
@jbweston jbweston deleted the feat/atomic-writes branch November 15, 2023 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

save_dataframe does not write atomically
2 participants