Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix google cloud storage with rustls + http2 #1985

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

saroh
Copy link

@saroh saroh commented Jun 8, 2022

Remove host http header when using rustls in order for storage.googleapis.com to function with rustls

Hi!
We ran into a error using Rusoto with rustls enabled (quickwit-oss/quickwit#1584). It turns out Google rejects the http2 payload if it contains a host header. This fix feels a little hacky given that google seems to be the only provider to act this way. I can close this PR and open an issue if you prefer :)

Sample code which errors on the current main branch:

use rusoto_core::Region;
use rusoto_s3::{S3Client, S3};

#[tokio::main]
async fn main() {
    let region = Region::Custom {
        name: "gcs".to_string(),
        endpoint: "https://storage.googleapis.com".to_string(),
    };
    let client = S3Client::new(region);
    println!("{:?}", client.list_buckets().await);
}

outputs Err(HttpDispatch(HttpDispatchError { message: "Error obtaining body: Error obtaining chunk: error reading a body from connection: stream error received: unspecific protocol error detected" }))

with this branch: Ok(ListBucketsOutput { buckets: Some([Bucket { creation_date: Some("2022-06-05T18:03:45.929Z"), name: Some("qw-region") }]), owner: None })

It's possible to use rusoto with rustls and GCP using a http1 client (quickwit-oss/quickwit#1612)

for storage.googleapis.com to function with rustls
@saroh saroh force-pushed the fix-google-cloud-storage branch from 0aba603 to 62e2017 Compare June 8, 2022 17:18
// we use HTTP/2 with rustls, the host header is required to sign the request
// Google cloud storage rejects HTTP/2 payloads containing a host header
#[cfg(any(feature = "rustls", feature = "rustls-webpki"))]
hyper_headers.remove("host");
Copy link
Author

@saroh saroh Jun 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Host header is required for HTTP/1.1 but not HTTP/2 it seems: https://datatracker.ietf.org/doc/html/rfc7540#section-8.1.2.3

It'd be safer/necessary to put this behind a http2 or gcp-http2 feature because this'd break custom clients using HTTP/1 imho.

@saroh saroh changed the title Fix google cloud storage with rustls Fix google cloud storage with rustls + http2 Jun 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant