Allow custom endpoints with Google Cloud Storage.

[martey]

Add new GS_CUSTOM_ENDPOINT setting to allow usage of custom domains/URLs with Google Cloud Storage, similar to AWS_S3_CUSTOM_DOMAIN.

[martey]

I've updated this commit so that it is compatible with google-cloud-storage 1.15.0 (the addition of support for v4 signed URLs broke one of the imports added here). Because v4 signed URLs are still in beta, this commit still uses v2 signed URLs.

I've updated the google-cloud-storage minimum version to 1.15.0, but I can change it to be backwards compatible if necessary.

[martey]

Hi @jschneier, can you let me know if there is anything else I should do to help this get merged?

[hiimdoublej]

+1 would like this as well @jschneier

[zjsvv]

Hi, @sww314 and @jschneier, we are using this PR on production for a couple of months, and it works well. Can you give some advices to help this get merged? Thanks.

[martey]

I've rebased this so that it can merge cleanly with master. I would still like this to be merged, and would appreciate feedback from this library's maintainers.

[andruten]

I'm also interested in this feature!

[martey]

I've updated this to fix a bug with generating signed URLs and to stop using django.utils.six.

[jschneier]

Is this change required?

[martey]

The code in this pull request relies on the fact that Blob.generate_signed_url has a api_access_endpoint parameter (introduced in googleapis/google-cloud-python#7460 and google-cloud-storage 1.15.0).

It would be theoretically possible to update the code so that it is still compatible with google-cloud-storage 0.22.0, but because of the substantial differences in google-cloud-storage pre- and post-1.15.0, this would take a substantial amount of work and would significantly increase the complexity of this pull request's code.

[jschneier]

That change was merged in April. I'm not sure if I can unilaterally require that upgrade.

Is the major difference that non-public URLs wouldn't be able to be signed?

[jschneier]

I see there's quite a bit of demand for this, what did a previous iteration look like that didn't mandate the version upgrade? I want to merge some variant of this.

[martey]

I see there's quite a bit of demand for this, what did a previous iteration look like that didn't mandate the version upgrade?

The "previous iteration" from January of this year (cb0c0c4) was created before google-cloud-storage 0.15.0 existed, and isn't compatible with any releases >= 0.15.0. It wouldn't be a valid option unless we wanted to require users have a version of google-cloud-storage before 0.15.0 or (as you mentioned in your earlier comment) were willing to prevent non-public URLs from working.

I am unsure about whether backporting these changes to google-cloud-storage < 0.15.0 would be successful. The version requirement on google-cloud-storage in setup.py encourages installing the most recent version of google-cloud-storage, so the automated tests don't cover using an older version of google-cloud-storage and it's possible that there are additional hidden bugs or problems in those older versions. It's also not clear that older versions like 0.22.0 are supported by Google (google-cloud-storage's changelog starts with version 1.6.0).

Personally, I don't think mandating a library upgrade is a huge ask for a new feature, especially when it would only affect people who have explicitly pinned google-cloud-storage. That said, I am still interested in having this merged, and will try to make whatever changes are necessary to have this happen.

[jschneier]

You make good points...especially that our version checking is loose see e.g the failing tests on the PR. That one would have been caught by running the tests every day though.

At some point I will need to rip the bandaid off and remove all of the Python 2 related and deprecated things. Since I have historically been willing to break backwards compat when I increment minor versions I don't think it's too much ask to require it for the next one but it does leave open a question of when that should be done.

Can you please rebase to fix the tests.

[martey]

I've rebased this and ensured that all tests pass.