Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace RClone sync by AWS CLI sync #5833

Open
5 tasks
Tracked by #1313
sanderegg opened this issue May 16, 2024 · 0 comments
Open
5 tasks
Tracked by #1313

Replace RClone sync by AWS CLI sync #5833

sanderegg opened this issue May 16, 2024 · 0 comments
Assignees
Labels
a:agent agent service a:dynamic-sidecar dynamic-sidecar service

Comments

@sanderegg
Copy link
Member

sanderegg commented May 16, 2024

After extensive testing in #5828 it showed that the AWS CLI is approximatively 1.5-2x faster at transferring data between local path and S3 than the current RClone sync used in dynamic-sidecar and agent.

It also is backed by AWS directly so should hopefuly be more stable (RClone 1.66.0 upgrade was a disaster as uploading would create kernel panics).

Since boto3 does not implement these tools, it would be needed to wrap the AWS CLI, checking whether there are any available tool already doing this might be worth the time.

Tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:agent agent service a:dynamic-sidecar dynamic-sidecar service
Projects
None yet
Development

No branches or pull requests

4 participants