[storage] Confusing behavior when using a custom endpoint

[megaserg]

Does this issue affect the google-cloud-cpp project?
I believe so.

What component of google-cloud-cpp is this related to?
GCS/storage/CURL client.

Describe the bug

• TL;DR: Tried to use a custom endpoint, some requests properly go to the endpoint, and some go to storage.googleapis.com instead. This is because CURL client 1) automatically switches to XML API unless certain headers are provided 2) ignores the custom endpoint for XML requests in favor of CLOUD_STORAGE_TESTBENCH_ENDPOINT.

• Long version: I am trying to add GCS support in apache/arrow. I've built and installed the google-cloud-cpp project as described in doc/packaging.md (on Ubuntu 18.04), except that I made shared libs.

• Then, I wrote some C++ code implementing a filesystem interface for Arrow, using google::cloud::storage client.

• Then, I wrote a test using fsouza/fake-gcs-server. The test setup creates a bucket, creates a file, and reads the file back.

• To be able to use fake-gcs-server, I launch the server like this:

docker run -i --rm -p 4443:4443 fsouza/fake-gcs-server:latest -scheme http

and provide a custom endpoint like this:

namespace gcs = google::cloud::storage;

StatusOr<gcs::ClientOptions> client_config_;
std::unique_ptr<gcs::Client> client_;

client_config_ = gcs::ClientOptions::CreateDefaultClientOptions();
std::string endpoint = "http://127.0.0.1:4443";
client_config_->set_endpoint(endpoint);
auto creds = gcs::oauth2::CreateAnonymousCredentials();
client_config_->set_credentials(creds);
client_.reset(new gcs::Client(*client_config_));

• Then, using gcs::Client, I create a bucket bucket, insert an object somefile, and read an object with client_->ReadObject("bucket", "somefile");. The resulting gcs::ObjectReadStream has a badbit, and the status just says Permanent error in Read(): [PERMISSION_DENIED].
• It turns out, the google-cloud-cpp's CURL client
  • automatically switches to XML API unless certain headers are provided: https://github.com/googleapis/google-cloud-cpp/blob/v1.17.0/google/cloud/storage/internal/curl_client.cc#L566
  • ignores the custom-provided endpoint for XML requests in favor of CLOUD_STORAGE_TESTBENCH_ENDPOINT: https://github.com/googleapis/google-cloud-cpp/blob/v1.17.0/google/cloud/storage/internal/curl_client.cc#L242
  • as a result, the upload request went to my custom endpoint, but the download request went to the real Google Storage API endpoint, resulting in a cryptic "PERMISSION_DENIED" error.

Expected behavior

• I'm actually not sure how serious this is and what I would expect 😅. In my usecase, fake-gcs-server doesn't support XML API. I guess the expected behavior here would be to stick to custom endpoint even if switched to XML API, and then I could see the unsupported web server route error. Or at least I'd like the error message to be informative enough that I can understand that the wrong endpoint is being used.

Operating system:
Ubuntu 18.04.2 LTS
Linux 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

What compiler and version are you using?
c++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0

What version of google-cloud-cpp are you using?
Git tag v1.17.0.

[coryan]

nI'm actually not sure how serious this is and what I would expect 😅. In my usecase, fake-gcs-server doesn't support XML API. I guess the expected behavior here would be to stick to custom endpoint even if switched to XML API,

Or maybe disable switching to the XML API if any custom endpoint is being used.

and then I could see the unsupported web server route error. Or at least I'd like the error message to be informative enough that I can understand that the wrong endpoint is being used.

I agree, but I think disabling the switch is cleaner. What do you think?

As a workaround, can you use the testbench included in google/cloud/storage/testbench ?

[megaserg]

Yes, disabling the switch will work too. I shall try the testbench!

[coryan]

Or maybe disable switching to the XML API if any custom endpoint is being used.

Yes, disabling the switch will work too.

That, unfortunately, will not work. Some folks override the endpoint to use private interconnects, and we would want to keep XML enabled in that case. I see that fsouza/fake-gcs-server has at least partial support for XML, maybe we can use that.

[megaserg]

Hmm why do you think it has partial XML support? I didn't find anything in code or HTTP routes.

[coryan]

This looks like fsouza/fake-gcs-server setting an XML handler to me:

https://github.com/fsouza/fake-gcs-server/blob/bb30ba529ac5a206f8a9fa3bf77b15cf12c2a840/fakestorage/server.go#L221

But I have not looked further to see if operations to PUT/POST/GET objects behave as the service does, e.g., they parse and return the right HTTP headers.

[coryan]

Okay, I looked further, definitely no support for XML, filed fsouza/fake-gcs-server#331

[coryan]

And also fsouza/fake-gcs-server#332

[coryan]

I have setup things to automatically close this issue via #5100. Once that change is in there are no actionable things for us to do, there are some problems using the fake-gcs-server, but those are not fixable (or should not be fixed) in the library.