Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: add background task to periodically clean up data in S3 #4673

Open
1 task done
sanderegg opened this issue Aug 28, 2023 · 0 comments
Open
1 task done

storage: add background task to periodically clean up data in S3 #4673

sanderegg opened this issue Aug 28, 2023 · 0 comments
Assignees
Labels
a:storage issue related to storage service t:maintenance Some planned maintenance work

Comments

@sanderegg
Copy link
Member

Is there an existing issue for this?

  • I have searched the existing issues

Planned changes

Since we do now have folders inside S3, the synchronisation between postgres and S3 is most probably broken in at least the following use-cases:

  • deleting folders and storage is restarted, or dead-locked, etc... --> S3 files are lying there forever which implies costs

Therefore it is desirable to have a background task that runs with low priority and checks that the S3 contents matches the database, for instance:

  • any /node_id that does not exist in the db should be removed
  • any /project_id that does not exist in the db should be removed
  • also be careful of current uploads

This is a sub issue from ITISFoundation/osparc-issues#675

@sanderegg sanderegg added a:storage issue related to storage service t:maintenance Some planned maintenance work labels Aug 28, 2023
@sanderegg sanderegg added this to the Baklava milestone Aug 28, 2023
@pcrespov pcrespov modified the milestones: Baklava, the nameless Sep 13, 2023
@sanderegg sanderegg removed this from the the nameless milestone Apr 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:storage issue related to storage service t:maintenance Some planned maintenance work
Projects
None yet
Development

No branches or pull requests

4 participants