Handle 410 errors on resumable-media operations

[crwilcox]

A previous issue googleapis/google-cloud-python#7530 was closed in favor of a targeted issue for a feature request.

Resumable media operations can fail in such a way that they cannot be retried, at least on the chunk level. A 410 error indicates that the only choice is to restart the operation altogether

Pseudo code:

try:
   resumable_operation_upload(some_file)
except 410_error:
    # Retry the operation, from the very beginning of the file.

We should find the instances of resumable uploads and protect them from the higher level failure. It is also possible this could be pushed down into resumable media, but these higher level failures are a different category than existing retry-able errors.

Related: https://issuetracker.google.com/137168102 and internal bug 115694647.

[andrewsg]

Looking at resumable media retries this week. I'm confused by this bug. Under what circumstances will the API issue a 410? That seems like a bizarre error code to issue in response to an upload request.

[crwilcox]

@andrewsg in practice, our front ends/services can sporadically send a variety of errors, notably unavailable (503) or gone (410). Users (our libraries) should understand how to reattempt.

For the 410 case, an example could be a situation where you are trying to upload a file, but somehow it is deleted, or corrupt. Gone will presumably be returned.

[frankyn]

@andrewsg do you think this is still necessary for us to do? I'm hesitant because 410 tells us the resumable upload session was either Deleted by the user or the service (happens after a week). 410 outside of that shouldn't happen and would be a service bug. Additionally, it's difficult to retry a resumable upload from the beginning if we don't have all the data available.