Signed url upload returns 200 response instead of 308 for headers = {"Content-Length": "0", "Content-Range": "bytes /"} #1362

j-antunes · 2023-10-12T17:12:22Z

Hello!

I think for the resumable uploads there is a small difference between the emulated and real GCS for when we start an upload.

The below code snippet is an example of an initial request that we can do to get the current status of the upload. You usually do this to see if there is anything already uploaded.

import requests
headers = {"Content-Length": "0", "Content-Range": "bytes */*"}
response = requests.put(session_uri, headers=headers)

I would expect to receive a 308 response (upload incomplete), but instead I'm getting a 200 response.

Note that you can skip this step and just directly upload the file, but it would be nice that this would work as expected.

Here is an example to reproduce the issue


# Copyright 2019 Francisco Souza. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.

# How to run this example
# 1 - Build the docker image by running the command "docker build -t fsouza/fake-gcs-server ."
# 2 - Start the docker container: "docker run -d --name fake-gcs-server -p 4443:4443 -v ${PWD}/examples/data:/data fsouza/fake-gcs-server -scheme http"
# 3 - Check if it's working by running: "curl http://0.0.0.0:4443/storage/v1/b"
# 4 - Create a python virtual enviroment (Ex: python -m .venv venv)
# 5 - Source the env (source .venv/bin/activate)
# 7 - Go to the following directory examples/python: (cd examples/python)
# 6 - Install requirements: "pip install -r requirements.txt" and "pip install -r requirements.in"
# 7 - Run this script

import tempfile

from google.auth.credentials import AnonymousCredentials
from google.cloud import storage
import requests

client = storage.Client(
    credentials=AnonymousCredentials(),
    project="test",
    # This endpoint assumes that you are using the default port 4443 from the container.
    # If you are using a different port you need to update this endpoint
    client_options={"api_endpoint": "http://localhost:4443"},
)

bucket = client.bucket("sample-bucket")
with tempfile.NamedTemporaryFile(suffix=".html", delete=False) as temp_file:
    temp_file.write(b"<html><body><h1>Hello, World!</h1></body></html>")

    blob = bucket.blob(temp_file.name)

    session_url = blob.create_resumable_upload_session(client=client)


headers = {"Content-Length": "0", "Content-Range": "bytes */*"}
response = requests.put(session_url, headers=headers)

print(response.json())

This is the response that I got

<Response [200]>
special variables
function variables
apparent_encoding:
'ascii'
connection:
<requests.adapters.HTTPAdapter object at 0x7f2d951ed6f0>
content:
b'{"kind":"storage#object","name":"/tmp/tmpmsqjyo4o.html","id":"sample-bucket//tmp/tmpmsqjyo4o.html","bucket":"sample-bucket","size":"0","contentType":"application/octet-stream","crc32c":"AAAAAA==","acl":[{"bucket":"sample-bucket","entity":"projectOwner-test-project","object":"/tmp/tmpmsqjyo4o.html","projectTeam":{},"role":"OWNER"}],"md5Hash":"1B2M2Y8AsgTpgAmY7PhCfg==","etag":"\\"1B2M2Y8AsgTpgAmY7PhCfg==\\"","timeCreated":"2023-10-12T17:03:59.99945Z","updated":"2023-10-12T17:03:59.999452Z","generation":"1697130239999453"}\n'
cookies:
<RequestsCookieJar[]>
elapsed:
datetime.timedelta(microseconds=2909)
encoding:
'utf-8'
headers:
{'Content-Type': 'application/json', 'Range': 'bytes=0-0', 'Date': 'Thu, 12 Oct 2023 17:04:00 GMT', 'Content-Length': '524'}
history:
[]
is_permanent_redirect:
False
is_redirect:
False
links:
{}
next:
None
ok:
True
raw:
<urllib3.response.HTTPResponse object at 0x7f2d951eec80>
reason:
'OK'
request:
<PreparedRequest [PUT]>
status_code:
200
text:
'{"kind":"storage#object","name":"/tmp/tmpmsqjyo4o.html","id":"sample-bucket//tmp/tmpmsqjyo4o.html","bucket":"sample-bucket","size":"0","contentType":"application/octet-stream","crc32c":"AAAAAA==","acl":[{"bucket":"sample-bucket","entity":"projectOwner-test-project","object":"/tmp/tmpmsqjyo4o.html","projectTeam":{},"role":"OWNER"}],"md5Hash":"1B2M2Y8AsgTpgAmY7PhCfg==","etag":"\\"1B2M2Y8AsgTpgAmY7PhCfg==\\"","timeCreated":"2023-10-12T17:03:59.99945Z","updated":"2023-10-12T17:03:59.999452Z","generation":"1697130239999453"}\n'
url:
'http://0.0.0.0:4443/upload/storage/v1/b/sample-bucket/o?uploadType=resumable&name=%2Ftmp%2Ftmpmsqjyo4o.html&upload_id=7e7cd5bec6cf35ee112a0f99e4b57f2b'
_content:
b'{"kind":"storage#object","name":"/tmp/tmpmsqjyo4o.html","id":"sample-bucket//tmp/tmpmsqjyo4o.html","bucket":"sample-bucket","size":"0","contentType":"application/octet-stream","crc32c":"AAAAAA==","acl":[{"bucket":"sample-bucket","entity":"projectOwner-test-project","object":"/tmp/tmpmsqjyo4o.html","projectTeam":{},"role":"OWNER"}],"md5Hash":"1B2M2Y8AsgTpgAmY7PhCfg==","etag":"\\"1B2M2Y8AsgTpgAmY7PhCfg==\\"","timeCreated":"2023-10-12T17:03:59.99945Z","updated":"2023-10-12T17:03:59.999452Z","generation":"1697130239999453"}\n'
_content_consumed:
True
_next:
None

Running the same example against the real GCS I get the following response:

<Response [308]>
special variables
function variables
apparent_encoding:
None
connection:
<requests.adapters.HTTPAdapter object at 0x7f396620d0f0>
content:
b''
cookies:
<RequestsCookieJar[]>
elapsed:
datetime.timedelta(microseconds=147664)
encoding:
'utf-8'
headers:
{'Content-Type': 'text/plain; charset=utf-8', 'X-GUploader-UploadID': '<REDACTED>', 'Content-Length': '0', 'Date': 'Thu, 12 Oct 2023 17:08:36 GMT', 'Server': 'UploadServer', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'}
history:
[]
is_permanent_redirect:
False
is_redirect:
False
links:
{}
next:
None
ok:
True
raw:
<urllib3.response.HTTPResponse object at 0x7f396620c820>
reason:
'Resume Incomplete'
request:
<PreparedRequest [PUT]>
status_code:
308
text:
''
url:
'https://storage.googleapis.com/upload/storage/v1/b/<REDACTED>/o?uploadType=resumable&upload_id=<REDACTED>'
_content:
b''
_content_consumed:
True
_next:
None

The text was updated successfully, but these errors were encountered:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Signed url upload returns 200 response instead of 308 for headers = {"Content-Length": "0", "Content-Range": "bytes /"} #1362

Signed url upload returns 200 response instead of 308 for headers = {"Content-Length": "0", "Content-Range": "bytes /"} #1362

j-antunes commented Oct 12, 2023

Signed url upload returns 200 response instead of 308 for headers = {"Content-Length": "0", "Content-Range": "bytes */*"} #1362

Signed url upload returns 200 response instead of 308 for headers = {"Content-Length": "0", "Content-Range": "bytes */*"} #1362

Comments

j-antunes commented Oct 12, 2023

Signed url upload returns 200 response instead of 308 for headers = {"Content-Length": "0", "Content-Range": "bytes /"} #1362

Signed url upload returns 200 response instead of 308 for headers = {"Content-Length": "0", "Content-Range": "bytes /"} #1362