Fix object_store multipart uploads on S3 Compatible Stores #2731
Conversation
The official Minio SDK uses "uploads=" as the URL when it initiates a multipart upload instead of "uploads". This affects the AWSV4 signature and causes object_store to fail a signature check when initiating the upload to Minio. It's possible that this contradicts the AWS S3 API docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html#API_CreateMultipartUpload_RequestSyntax and we need to instead keep the URL as `?uploads` and change the URL that goes into the signature instead.
tustvold
changed the title
|
Thank you 😄 |
|
Benchmark runs are scheduled for baseline = fb01656 and contender = 5f441ee. 5f441ee is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
I couldn't find an issue for this in the arrow-rs repo. Some investigation here: splitgraph/seafowl#112
Rationale for this change
The official Minio SDK uses
/?uploads=as the URL when it initiates a multipart upload instead of/?uploads(source). This affects the query param string that goes into the AWSV4 signature and causes object_store to fail a signature check when initiating a multipart upload to Minio.NOTE: This is a deviation from the AWS S3 REST API docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html#API_CreateMultipartUpload_RequestSyntax (which do use
/?uploads). I haven't tested that this doesn't now break S3 multipart uploads.As an alternative, we could instead alter the query param string in the signature itself, though from a very quick glance at the Minio source code it looks like it uses the query string from the original request URL.
What changes are included in this PR?
Change the URL to initiate a multipart upload to have
?uploads=instead of?uploads.Are there any user-facing changes?
None