Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mc mirror takes long time when target bucket contains a lot of files #4873

Open
archsh opened this issue Mar 14, 2024 · 2 comments
Open

mc mirror takes long time when target bucket contains a lot of files #4873

archsh opened this issue Mar 14, 2024 · 2 comments
Assignees

Comments

@archsh
Copy link

archsh commented Mar 14, 2024

Expected behavior

Using mc mirror to mirror local folder to a target bucket, it should be exit soon if local folder is empty or few files.

Actual behavior

It took a long time to start transfer files if the target bucket contains a lot of objects.

Steps to reproduce the behavior

mc mirror --json ./localfolder target/bucket/path

mc --version

  • (paste output of mc --version)
    mc version RELEASE.2024-01-05T05-04-32Z (commit-id=59eca9fea8984adec1e8e7a1c95d0ea23107ceff)

System information

CentOS release 6.9 (Final) x64

@zveinn
Copy link
Contributor

zveinn commented Mar 15, 2024

Yes, mc mirror does a full source to target comparison when it's ran. This is due to the --remove flag. However, it might be possible to speed up the mirror given that the --remove flag is not present. I will look into this in the coming weeks, no promises on delivery date though since my schedule is rather full.

@zveinn zveinn self-assigned this Mar 15, 2024
@PrisedRabbit
Copy link

PrisedRabbit commented May 24, 2024

Same here. It takes a lot of time to mirror folder with 20 000+ files ( using aws cli solved the problem )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants