Random MultipartUpload -> RequestError -> "use of closed network connection" when uploading a lot of data to S3

[segevfiner]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
We have code that download and uploads S3 objects across different buckets (We can't use CopyObject/UploadPartCopy as it is done from public buckets that are out of our control).

On larger buckets, after some time passes, we randomly get an error that looks something like this:

MultipartUpload: upload multipart failed
    upload id: <snip>
caused by: RequestError: send request failed
caused by: Put https://<snip>.s3.amazonaws.com/some_object?partNumber=4&uploadId=<snip>: write tcp 172.20.0.57:54788->52.217.14.236:443: use of closed network connection

The SDK is configured, as per default, to retry requests (I think the default for S3 is 3 retries). But it might not be retrying for this specific error. At least I wasn't able to find a reference to it in the SDK code. (I think this is one of those errors that Go hasn't exported for unknown reasons golang/go#4373)

I'm not sure if this is an error that can/should occur sporadically and should be retried by the SDK or it arises from some race condition/bug somewhere in the SDK or Go.

Version of AWS SDK for Go?
v1.31.15

Version of Go (go version)?
go version go1.13.12 linux/amd64

To Reproduce (observed behavior)
https://github.com/segevfiner/s3-download-upload-stress

There is one MWE there to copy an object from one bucket over and over to another, and one that copies a bucket recursively to another.

Expected behavior
The entire copy process should work to the end and not crash midway with a random error, the SDK should retry internally for transient errors.

Additional context
Add any other context about the problem here.

[diehlaws]

Hi @segevfiner, thanks for reaching out to us about this. It sounds like the SDK is attempting to re-use a connection that has been closed by S3. This should be solvable by implementing a custom HTTP client in your session's config with a Dialer.KeepAlive value lower than the default 30 seconds (or a negative value to disable keep-alives). Please do let us know if you continue to see this behavior when using a custom keep-alive value on the HTTP client used by the SDK for your S3 calls.

[github-actions]

This issue has not recieved a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[segevfiner]

(Commenting to stop auto close)

[praneetloke]

@diehlaws I followed your recommendation for setting the keep-alive interval value to less than 30s, but I still see the use of closed network connection error from the SDK during uploads. I followed the recommendation from the link you provided in your comment for creating custom HTTP clients.

Here's the relevant snippet. Note that the one difference is that I also set the Timeout property of the http.Client{} type (ClientTimeout setting in the below snippet) in addition to the settings on the http.Transport type.

// The values specified here are the default values defined in the net/http
// package's DefaultTransport instance, except where noted.
httpClient, err := NewHTTPClientWithSettings(HTTPClientSettings{
        // The HTTP Client total timeout value for the request.
	ClientTimeout: 30 * time.Second,

	Connect:               30 * time.Second,
        // Set the keep-alive interval to less than 30s.
	ConnKeepAlive:    10 * time.Second,
	ExpectContinue:   1 * time.Second,
	IdleConn:              30 * time.Second,

	MaxAllIdleConns:    100,
	MaxHostIdleConns: 2,

	ResponseHeader:   5 * time.Second,
	TLSHandshake:      10 * time.Second,
})

[matthewswain]

I've been looking at this as I've experienced the same behaviour.

It looks to me like:

• When the "use of a closed network connection" error is encountered, it's wrapped with awserr.New using ErrCodeRequestError (see here)
• This bubbles up to be evaluated by shouldRetryError in the retryer
  (see here)
• Which in turn passes it to isNestedErrorRetryable (see here)
• isNestedErrorRetryable doesn't seem to have provision for this particular error being considered retryable

The original error is defined here. Perhaps isNestedErrorRetryable should be returning true in this instance?

[matthewswain]

Scratch that. Adding the following, passing test case to TestIsErrorRetryable in aws/request/retryer_test.go seems to invalidate the theory that shouldRetryError returns false.

{
	Err:       awserr.New(ErrCodeRequestError, "send request failed", errors.New("use of closed network connection")),
	Retryable: true,
},

[jschaf]

We see this error multiple times per day uploading Postgres backups to S3 with many TBs of data. We have a failure rate of about 5% of our backups. The error manifests through wal-g, a Go library that uploads a Postgres backup to S3. Here's the wal-G stack trace:

The AWS sdk used in our version is v1.26.1.

ERROR: 2020/08/06 08:39:52.198782 failed to upload 'basebackups_005/base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br' to bucket 'S3_BUCKET':
    MultipartUpload: upload multipart failed
caused by: RequestError: send request failed
caused by: Put https://S3_BUCKET/basebackups_005/base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br?partNumber=2:
    write tcp 10.64.18.161:42118->52.216.134.19:443: use of closed network connection
ERROR: 2020/08/06 08:39:52.198805 upload: could not upload 'base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br'
ERROR: 2020/08/06 08:39:52.198818 failed to upload 'basebackups_005/base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br' to bucket 'S3_BUCKET':
    MultipartUpload: upload multipart failed
caused by: RequestError: send request failed
caused by: Put https://S3_BUCKET/basebackups_005/base_0000000100006F04000000E4/tar_partitions/part_19148.tar.br?partNumber=2
    write tcp 10.64.18.161:42118->52.216.134.19:443: use of closed network connection
ERROR: 2020/08/06 08:39:52.198833 Unable to complete uploads

[rowandh]

I think this issue may still exist on Windows machines. I'm using https://github.com/peak/s5cmd with aws-sdk-go v1.34.12 built for Windows. Everything is working fine but I receive this error very often: wsarecv: an existing connection was forcibly closed by the remote host.

It seems likely that this error message is unique to the Windows TCP/IP stack's WSAECONNRESET error code - see https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2. Because it's Windows-specific, it's being missed by the error string matching which only checks for a *nix message.

I'm not familiar with how Go handles error messages on different platforms so it's possible I'm beating down the wrong path with this. However adding additional handling for this error message in the retry logic seemed to fix the issue for me.

[larytet]

Happens in a Linux container
write tcp my_ip:51294->s3_ip:443: use of closed network connection
Should I upgrade the AWS SDK?

[bartdenotte]

We are recently seeing more occurrences of this issue. Is there a reason why this is ticket is open again?

[AMZN-hgoffin]

I believe the problem here is that the AWS SDK for Go is
1/ not forcing a new, non-reused connection for non-idempotent requests as HTTP spec strongly suggests (and which most browsers implement)
2/ is not treating connection closure as a retriable error for idempotent requests

The workaround is to disable HTTP transport keep-alive (which is NOT the same thing as Dialer keep-alive, that is unrelated to the problem), or set the idle connection timeout very short on the client side in the hope that you never get close to the actual server timeout.