Allow setting the chunk_size in Blob creation via the GS_BLOB_CHUNK_SIZE

[joahim]

When uploading a large file to the Google Cloud Storage, the generated request must fit into memory and can easily exhaust it. This can be avoided by specifying the chunk_size at Blob creation which then triggers the resumable upload that uploads the file in chunks and does not load the complete file into memory.

See: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/d3ef455b797b6960ed58f563d4b43d9dcc4c7364/storage/google/cloud/storage/blob.py#L770

[codecov-io]

Codecov Report

Merging #401 into master will decrease coverage by 0.05%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #401      +/-   ##
==========================================
- Coverage    76.1%   76.05%   -0.06%     
==========================================
  Files          11       11              
  Lines        1578     1566      -12     
==========================================
- Hits         1201     1191      -10     
+ Misses        377      375       -2

Impacted Files	Coverage Δ
storages/backends/gcloud.py	95.7% <100%> (+0.9%)	⬆️
storages/backends/s3boto.py	87.41% <0%> (-0.05%)	⬇️
storages/backends/s3boto3.py	86.54% <0%> (-0.05%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea5649f...ff84614. Read the comment docs.

[myimages]

I think this cloud be very easily done with simple python inheritance.

I have approached the same problem by creating custom_storages.py in my app and overriding the GoogleCloudFile init method.

and in my new storage class PublicImageUploadStorage , I have overridden the _save method to use my new ChunkedGoogleCloudFile class since it is the only method that has the w write flag.

sooner or later , you will end up overriding the class to add your own customization ( setting metadata , changing default acl, bucket name , upload files directly without saving them locally , etc ) , so it is better done now.

[joahim]

@myimages I am already subclassing storages.backends.gcloud.GoogleCloudStorage to have additional methods from the GoogleStorage available. However, chunked uploads are in my opinion often required (also, Google suggest this method for all Blobs bigger than 5MB) and should therefore be available out of the box.

[dummerbd]

I think this should definitely be included in this lib instead of requiring people to subclass both GoogleCloudFile and GoogleCloudStorage and then overriding both the _open and _save methods to use your file subclass. Its a lot of effort just to set something that should probably be set to a reasonable default (like 5MB).

[sww314]

@joahim Can you add documentation for this setting?

[jschneier]

Added docs in #757.