[object_store] Better Error Handling & Response Propagation

[matthewfollegot]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

First off, thank you for having a client that works seamlessly for the various cloud providers and a clean interface to go along with it! The problem I'm encountering is related to error messages returned by the client.

Currently, error responses are limited to a seemingly short summary of the entire error response from the store (e.g. S3).
Here's an example error

"Generic S3 error: Error after 0 retries in 30.16539508s, max_retries:10, retry_timeout:300s, source:error sending request for url (https://s3.ap-northeast-1.amazonaws.com/<bucket>/<key>): operation timed out"

In order to further debug these errors and do things such as opening a support ticket with S3, it would be extremely helpful to include the attributes listed in the S3 docs here, some examples being the error code, message, and especially the request ID.

Describe the solution you'd like

I'd love to see as much of the original error propagated to the caller.
A solution could be to update all relevant or just the Generic variant of the object_store::Error with the fields mentioned above and propagate the errors from the store to the caller.

Describe alternatives you've considered

N/A

Additional context

I'm observing a large amount of errors on S3 PUT requests.

[tustvold]

The provided error is a timeout not an error returned by AWS, in cases where an error is returned by AWS we print the response body. In this case there is no response body as the request took too long and hit a client side timeout before returning anything.

I would double check the ClientConfig is set appropriately for the size of uploads you are performing, and if you are uploading very large objects perhaps you might consider using a multipart upload - this will be faster and more reliable

[matthewfollegot]

The provided error is a timeout not an error returned by AWS, in cases where an error is returned by AWS we print the response body. In this case there is no response body as the request took too long and hit a client side timeout before returning anything.

I would double check the ClientConfig is set appropriately for the size of uploads you are performing, and if you are uploading very large objects perhaps you might consider using a multipart upload - this will be faster and more reliable

Thanks for the quick response! I have 10s of deployments running with the same client config and a maximum object size that is consistent across all deployments and doesn't change, and there's only a single deployment that's encountering this issue consistently, despite it being one of many staging/sandbox/shadow deployments, so I'm not too sure a multipart upload is absolutely necessary. I'll continue debugging and maybe try multipart uploads too just in case.