Experimental QUIC and HTTP 3 support in Istio gateways

QUIC and HTTP/3 support in Istio gateways

Istio provides alpha support for HTTP/3 at the gateways. Currently, it is restricted to using mirror TLS-terminated HTTPS servers at the gateway. For instance, if there is a TLS-terminated HTTPS server at 443/TCP, then an HTTP/3 server at 443/UDP is created so that clients which support QUIC (like Google Chrome) can use HTTP/3.

Setup

Prerequisites for the demo

• Kubernetes cluster with MixedProtocolLBService feature gate enabled.
• Set PILOT_ENABLE_QUIC_LISTENERS on the Pilot
• Assuming 443/TCP is exposed, expose 443/UDP on the gateway and allow 443/UDP on the firewall

Istio Operator config

As of writing this, this feature was still in the main, unreleased branch. So the image is built from this branch. Once v1.12 is released then setting hub and tag won't be necessary to use this feature.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: install
spec:
  hub: localhost:5000
  tag: master
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        service:
          ports:
          - name: status-port
            port: 15021
            targetPort: 15021
          # Both HTTPS and HTTP/3 must have
          # the same port number. Here it
          # is 443/TCP and 443/UDP
          - name: https
            port: 443
            targetPort: 8443
          - name: http3
            port: 443
            targetPort: 8443
            protocol: UDP
  values:
    pilot:
      # This is required to create mirror QUIC
      # listeners for TLS-terminated HTTPS listeners
      # on the gateway
      env:
        PILOT_ENABLE_QUIC_LISTENERS: true

Install with the above IstioOperator YAML

$ istioctl apply -f istio-operator.yaml -y
$ istioctl proxy-status

Testing

This demo uses the httpbin application under samples/ in istio/istio repository.

Deploying the application and setting up certificates

1. Create httpbin namespace and label it for istio injection

   $ kubectl create namespace httpbin
   $ kubectl label namespace httpbin istio-injection=enabled
   

2. Deploy httpbin application

   $ kubectl -n httpbin apply -f samples/httpbin/httpbin.yaml
   

3. Create TLS certificates for the ingress

   1. Configuration for the certificate

      [req]
      default_bits       = 2048
      prompt             = no
      distinguished_name = req_distinguished_name
      req_extensions     = san_reqext
      
      [ req_distinguished_name ]
      countryName         = IN
      stateOrProvinceName = KA
      organizationName    = QuicCorp
      
      [ san_reqext ]
      subjectAltName      = @alt_names
      
      [alt_names]
      DNS.0   = httpbin.quic-corp.com
      

   2. Generate CA and server certificates

      $ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:4096 -subj "/C=IN/ST=KA/O=QuicCorp" -keyout quiccorp-ca.key -out quiccorp-ca.crt
      
      $ openssl req -out httpbin.csr -newkey rsa:2048 -nodes -keyout httpbin.key -config httpbin.cnf
      
      $ openssl x509 -req -days 365 -CA quiccorp-ca.crt -CAkey quiccorp-ca.key -set_serial 0 -in httpbin.csr -out httpbin.crt -extfile httpbin.cnf -extensions san_reqext
      

   3. Create a secret in istio-system namespace

      $ kubectl -n istio-system create secret tls httpbin-cred --key=httpbin.key --cert=httpbin.crt
      

Istio configs to access with both HTTP/2(over TCP) and HTTP/3(over QUIC)

Gateway to expose the service (notice that there is no HTTP/3 or QUIC related configs here. The mirror server is created automatically)

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - httpbin.quic-corp.com
    tls:
      mode: SIMPLE
      credentialName: httpbin-cred

For routing to httpbin, we also need a VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-route
spec:
  hosts:
  - httpbin.quic-corp.com
  gateways:
  - httpbin-gateway
  http:
  - name: httpbin-default-route
    route:
    - destination:
        host: httpbin.httpbin.svc.cluster.local
        port: 
          number: 8000

Now apply, the configs and check if "two" listeners are created at the gateway for the specified port (one TCP and another QUIC listeners).

$ kubectl -n istio-system apply gateway-config.yaml

$ istioctl proxy-config listeners istio-ingressgateway-6fdcddbb8b-9rpkx.istio-system
ADDRESS PORT  MATCH                      DESTINATION
0.0.0.0 8443  SNI: httpbin.quic-corp.com Route: https.443.https.httpbin-gateway.istio-system
0.0.0.0 8443  SNI: httpbin.quic-corp.com Route: https.443.https.httpbin-gateway.istio-system
0.0.0.0 15021 ALL                        Inline Route: /healthz/ready*
0.0.0.0 15090 ALL                        Inline Route: /stats/prometheus*

You can examine the config dump further to check for listeners with transport socket of type envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport.

Testing with some traffic

NOTE: For testing HTTP/3 traffic a custom build of curl with HTTP/3 support may be required. In this demo it is called qcurl. Follow the instructions here.

First, note down the ingress gateway IP address as INGRESS_IP

$ kubectl -n istio-system get svc
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                       AGE
istio-ingressgateway   LoadBalancer   10.96.112.53    172.18.200.1   15021:32534/TCP,443:31597/TCP,443:31597/UDP   79m
istiod                 ClusterIP      10.96.255.123   <none>         15010/TCP,15012/TCP,443/TCP,15014/TCP         79m

$ export INGRESS_IP=172.18.200.1

Now, send HTTP/2 traffic (-k flag is used to skip TLS certificate verification. This is for demo purposes only)

$ curl -svk --http2 --resolve httpbin.quic-corp.com:443:$INGRESS_IP https://httpbin.quic-corp.com/headers
[ Output truncated ]
...
> GET /headers HTTP/2
> Host: httpbin.quic-corp.com
> user-agent: curl/7.76.1
> accept: */*
...
< HTTP/2 200 
< server: istio-envoy
< date: Wed, 27 Oct 2021 05:05:59 GMT
< content-type: application/json
< content-length: 601
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 1
< alt-svc: h3=":443"; ma=86400
< 
{
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.quic-corp.com", 
    "User-Agent": "curl/7.76.1", 
    "X-B3-Parentspanid": "819c2b2cab59bbe8", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "0462ac631afb5f84", 
    "X-B3-Traceid": "3fa44e3bbbcd21a3819c2b2cab59bbe8", 
    "X-Envoy-Attempt-Count": "1", 
    "X-Envoy-Internal": "true", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/httpbin/sa/httpbin;Hash=97bf9c90d4a5b9bb8f5da3e825dfa34f04631400420649394741807a320aa0a1;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }
}

Notice the alt-svc header in the reply. This may be used by the clients (like Google Chrome) to detect HTTP/3 support at the endpoint.

Now, send HTTP/3 traffic

$ qcurl -svk --http3 --resolve httpbin.quic-corp.com:443:$INGRESS_IP https://httpbin.quic-corp.com/headers
* Added httpbin.quic-corp.com:443:172.18.200.1 to DNS cache
* Hostname httpbin.quic-corp.com was found in DNS cache
*   Trying 172.18.200.1:443...
* Connect socket 5 over QUIC to 172.18.200.1:443
* Sent QUIC client Initial, ALPN: h3,h3-29,h3-28,h3-27
* Connected to httpbin.quic-corp.com () port 443 (#0)
* h3 [:method: GET]
* h3 [:path: /headers]
* h3 [:scheme: https]
* h3 [:authority: httpbin.quic-corp.com]
* h3 [user-agent: curl/7.78.0-DEV]
* h3 [accept: */*]
* Using HTTP/3 Stream ID: 0 (easy handle 0xa64260)
> GET /headers HTTP/3
> Host: httpbin.quic-corp.com
> user-agent: curl/7.78.0-DEV
> accept: */*
> 
< HTTP/3 200
< server: istio-envoy
< date: Wed, 27 Oct 2021 05:08:46 GMT
< content-type: application/json
< content-length: 642
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 2
< alt-svc: h3=":443"; ma=86400
< 
{
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.quic-corp.com", 
    "Transfer-Encoding": "chunked", 
    "User-Agent": "curl/7.78.0-DEV", 
    "X-B3-Parentspanid": "d9f3d78e3f6ddc5c", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "3f6c920b02a92b73", 
    "X-B3-Traceid": "d5c22cd31ddec119d9f3d78e3f6ddc5c", 
    "X-Envoy-Attempt-Count": "1", 
    "X-Envoy-Internal": "true", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/httpbin/sa/httpbin;Hash=97bf9c90d4a5b9bb8f5da3e825dfa34f04631400420649394741807a320aa0a1;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }
}