[request]: S3 multipart upload - generate presigned urls for each part

[ZelCloud]

A note for the community

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue, please leave a comment

Tell us about your request

I would like to have a way to generate pre-signed urls for each multipart upload portion so the uploads can be done client side instead of server side.

If we have the basic setup like so

let region_provider = RegionProviderChain::default_provider().or_else("us-east-1");
let config = aws_config::from_env().region(region_provider).load().await;
let client = Client::new(&config);

let bucket = "bucket";
let key = "Test.docx";

let purl = client.create_multipart_upload()
		.bucket(bucket)
		.key(key)
		.send()
		.await;

We can create the multipart_upload and get the upload_id, but there doesn't seem to be a way to generate a pre-signed url for the parts now. The only option it seems is to upload the parts directly from the server. Using functions like:

// Something like so
let part_upload = client.upload_part()
	.body(some_data)
	.bucket(bucket)
	.key(key)
	.part_number(1)
	.upload_id(upload_id)
	.send()
	.await;
	
// While I'd like for something like
let upload_part_uris = client.upload_part()
	.bucket(bucket)
	.key(key)
	.part_numbers(3) // or whatever the part amount should be
	.upload_id(upload_id)				
	.presigned(PresigningConfig::expires_in(core::time::Duration::from_secs(60)).unwrap())
	.await;

Tell us about the problem you're trying to solve.

I'd like to generate pre-signed urls to do client side multipart upload. This is important to us, so we can achieve the following:

1. Able to show upload progress to the user (progress bar / percent)
2. Retry uploading a specific part vs forcing the user to retry uploading a massive file ( > 5Gb ) in the case of failure or hiccup.

This (to my knowledge) can't be achieve with

client.put_object()
	.bucket(bucket)
	.key(key)
	.presigned(PresigningConfig::expires_in(core::time::Duration::from_secs(60)).unwrap())
	.await;

Since this generates a put url, and that doesn't provide on upload progress information, it's not tolerant to failure, and there is a 5GB max file size limit with this vs the 5TB with multipart.

Are you currently working around this issue?

Not being able to provide upload progress to the user is a blocking feature, and we don't have a workaround with the aws-rust-sdk yet.

Additional context

No response

[rcoh]

we're working on it! thanks for reporting

[rcoh]

I have this working—I just want to note that you'll need to manually capture the ETag of each presigned URL response and provide them in your complete_multipart_upload call. The ETag is in the ETag header and 0 is not a valid part number (they must start at 1)

[rcoh]

a fix was merged, this is now usable in the next branch (or it will be in a couple of hours)

[dependencies]
aws-config = { git = "https://github.com/awslabs/aws-sdk-rust", branch = "next" }
aws-sdk-s3 = { git = "https://github.com/awslabs/aws-sdk-rust", branch = "next" }

[ZelCloud]

Hey @rcoh, I had the chance to try out the next branch. I've successfully managed to get the presigned urls and work out the upload progress. Thank you and the team for getting this feature out.

I'll test out part failure, and uploading a file over 5Gb a bit later and report back if there's anything of note on either of those 2 points.

Also an aside question (probably nothing to do with this sdk itself), but I'm unable to put or get a file from a presigned url in production, but localhost works fine. I assume I'm missing some kind of configuration setting, for the bucket. I do have the CORS setup with "Allowheaders", "AllowedMethods", "AllowedOrigins" with the allow all (*), Would I need to decreases the public access blocks?

[jdisanti]

This was released with v0.9.0.

Also an aside question (probably nothing to do with this sdk itself), but I'm unable to put or get a file from a presigned url in production, but localhost works fine. I assume I'm missing some kind of configuration setting, for the bucket. I do have the CORS setup with "Allowheaders", "AllowedMethods", "AllowedOrigins" with the allow all (*), Would I need to decreases the public access blocks?

The presigned URLs should definitely work with public access blocked. I don't have any idea what could be going wrong without more information though.

[ZelCloud]

Hey @jdisanti, everything works as expected in v0.9.0.

As for the presigned issue that was a fault fully on our side. There was a small spelling mistake in the aws_secret_key environment variable. No issues with the sdk itself.

Closing the issue since everything in the original comment feature list has been implemented and is working as expected.

Again thank you and the team for getting this out!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.