Google cloud storage v1

[Zatte]

A patch to add google cloud storage support to Afero
using GCS strong consistency guarantees to implement almost all afero intefaces.

V1 scope limitations:

• No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
• No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
• NOTE: Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.
• File modes will always bee 0755/0664 for folders/files when read back

Performance implications

• Sequential reads are performant
• Sequential writes are performant.
• Seek + Read or ReadAt is performant after the initial seek.
• Alternating reads/writes to the same file handler are highly inefficient. To get consistent FS behavior using an API that separates readers and writers we close any open readers before an write as well close open writers before a read (ensure the data is committed).
• Seek + Write such as WriteAt, Truncate, Seek+Write will work as expected but with significant overhead. Doing a seek + write will in effect download the old file/object, overlay it with the new writes and save it back. This is done in a streaming fashion so large files will not clog the memory but will trigger a full download and upload of the file/object.

How to test:

• Ensure you have a service account setup and $GOOGLE_APPLICATION_CREDENTIALS pointing to that file. Other methods for creating a client are supported by manually creating the gcloud client and using NewGcsFs(...) instead of NewGcsFsFromDefaultCredentials(...) as below
• Update afero_test.go

var gcsFs = NewGcsFsFromDefaultCredentials(context.Background(), "[GCS_BUCKET_USED_FOR_TESTING]", string(filepath.Separator))
var Fss = []Fs{&MemMapFs{}, &OsFs{}, gcsFs }

[CLAassistant]

All committers have signed the CLA.

[ivucica]

It would be interesting to see GCS support merged.

[lingtorp]

Do we have a ETA or timeline when this could be merged in? 😃

[0xmichalis]

Ever since this PR was submitted, afero has been updated to use go modules so @Zatte will need to rebase on top of current master, update to use go modules, and ensure the CI passes before we can merge this.

[Zatte]

Yeah, 2 years is some time for sure. I fear that going through will trigger a rewrite (my past self tends to be a worse programmer than current self :) ).

Before jumping into doing it I'm curious if the maintainers will ever merge this PR given that it pulls in 3rd party libs and the integration-tests require a valid google-svc-account to run. If not i could update the code to reside outside afero (and just implement the afero.Fs interface)

[0xmichalis]

Before jumping into doing it I'm curious if the maintainers will ever merge this PR given that it pulls in 3rd party libs

Both the use of the GCS SDK and the iterator library looks fine to me.

and the integration-tests require a valid google-svc-account to run.

cc @spf13

[lesichkovm]

3 years waiting to be merged?

[0xmichalis]

Closing in favor of #296 which is a more up-to-date stab at adding GCS support in afero.