client.upload_file is very slow only in the last five percent

[Qianhongbo]

I use boto3 client.upload_file to upload different sizes of files to s3, but I found that when I run my program for several hours the speed will drop especially in the last 5-10 percent. The beginning uploading speed is normal, but the last part of it will drop to a very low speed. I really can't figure out what is the root cause of this. Thanks for any suggestions.

def upload_file(self, bucket_name: str, remote_path: str, local_path: str,
                    verbose: bool = True, callback = None) -> None:
        """Upload file to the bucket

        Args:
            bucket_name (str): The bucket name
            remote_path (str): The file path in the bucket
            local_path (str): The local file path

        Raise Exception:
            Exception: Failed to upload file
        """
        assert is_file_exist(local_path), f"The local file {local_path} not exist"

        config = TransferConfig(
            multipart_threshold=1024 * 1024 * 8,  # 10MB
            max_concurrency=5,
            multipart_chunksize=1024 * 1024 * 8,  # 10MB
            use_threads=True
        )

        try:
            if verbose:
                file_size = os.path.getsize(local_path)
                print(f"Uploading: {local_path}")
                with tqdm(total=file_size, unit="bytes", unit_scale=True,
                          unit_divisor=1024, desc=f"Progress: ",
                          ncols=PROGRESS_BAR_WIDTH) as bar:
                    self.s3_client.upload_file(
                        local_path, bucket_name, remote_path, Config=config, Callback=bar.update)
            else:
                self.s3_client.upload_file(local_path, bucket_name, remote_path, Config=config)
            if callback is not None:
                callback(bucket_name, remote_path, local_path)
            return os.path.join(bucket_name, remote_path)
        except Exception as e:
            logger.error(f"Failed to upload file {local_path}. {e}")
            raise e

[RyanFitzSimmonsAK]

Hi @Qianhongbo, thanks for reaching out. I'm going to close this discussion in favor of the issue (#3893) that I replied to, as it's not clear that this is a guidance question (it could be a genuine bug). Thanks!