Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multipart concurrent chunk upload is inefficient. #2817

Closed
arunj-kp opened this issue Mar 3, 2022 · 1 comment
Closed

Multipart concurrent chunk upload is inefficient. #2817

arunj-kp opened this issue Mar 3, 2022 · 1 comment
Assignees
Labels
pending-triage Issue is pending triage s3 Issues with the AWS Android SDK for Simple Storage Service (S3).

Comments

@arunj-kp
Copy link
Contributor

arunj-kp commented Mar 3, 2022

Describe the bug
This PR(2804) unblocked us to try concurrent multipart upload and it is a great change. But, unfortunately, there is a bug in this feature. Redundant part uploads are happening. I uploaded a 7.8MB file with two parts (Part1: 5MB, Part2: 2.8MB) as I opted for the smallest chunk size of 5MB. But, AWS SDK uploaded five parts like 2(Part2),1(Part1),2,1,2 (That is, Part2 uploaded three times and Part1 two times) [Screenshot of the part upload sequence attached below]

To Reproduce

  • Enable more number of threads (I set 10 threads)
  • Upload a file (I uploaded 7.8MB file)
  • Monitor network traffic. I used CharlesProxy, I could see concurrent part uploads happening but keeps on uploading already uploaded parts
  • Finally succeeds
    This is a major performance issue in this feature and the feature is inefficient to use in a production app (uploaded around 20MB for a 7.8MB file)

A code sample or steps:

        val awsTempCredentials = BasicSessionCredentials(
            uploadKeyMap[ACCESS_KEY], uploadKeyMap[SECRET_KEY], uploadKeyMap[SESSION_TOKEN]
        )
        val awsClient = AmazonS3Client(awsTempCredentials, ClientConfiguration())
        awsClient.setS3ClientOptions(S3ClientOptions.builder().setAccelerateModeEnabled(true).build())
        val tuOptions = TransferUtilityOptions()
        tuOptions.transferThreadPoolSize = 1
        transferUtility = TransferUtility.builder()
            .s3Client(awsClient)
            .context(applicationContext)
            .transferUtilityOptions(tuOptions)
            .defaultBucket(uploadKeyMap[S3_BUCKET_NAME])
            .build()

            val uploadObserver = transferUtility?.upload(uploadKeyMap[UPLOAD_KEY], mediaFile)

            Single.create<TransferState> { emitter ->
                uploadObserver?.setTransferListener(UploadListener(emitter))
            }.blockingGet()

Which AWS service(s) are affected?
S3 file upload

Expected behavior
Redundant part uploads should not be happened. For a 7.8MB file, only 2 parts to be uploaded if the max part size is selected as 5MB.

Screenshots
If applicable, add screenshots to help explain your problem.

image

Environment Information (please complete the following information):

  • AWS Android SDK Version: 2.41.1
  • Device: Samsung J7
  • Android Version: 9
  • Specific to simulators: No

Additional context
Add any other context about the problem here.

@sdhuka sdhuka self-assigned this Mar 3, 2022
@sdhuka sdhuka added pending-triage Issue is pending triage s3 Issues with the AWS Android SDK for Simple Storage Service (S3). labels Mar 3, 2022
@sdhuka
Copy link
Contributor

sdhuka commented Mar 14, 2022

Discussed this offline with OP and not able to repro, if required please re-open.

@sdhuka sdhuka closed this as completed Mar 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pending-triage Issue is pending triage s3 Issues with the AWS Android SDK for Simple Storage Service (S3).
Projects
None yet
Development

No branches or pull requests

2 participants