Proxy Based Stackdriver Telemetry

Proxy Based Stackdriver Telemetry

Since 1.4, Istio supports exporting Stackdriver telemetry (metrics, logs, trace spans, and context graph) directly from proxy instead.

Metrics and Log

For version before 1.8 (< 1.8), to enable metrics and logs, add the following options into istioctl control plane configuration, or set them in istioctl command line:

values:
  telemetry:
    enabled: true
    v2:
      enabled: true
      stackdriver:
        enabled: true  # This enables Stackdriver metrics
        logging: true  # This enables Stackdriver server access logging

For version after 1.8 (>= 1.8), istio Stackdriver adds client side access log reporting, to enable logging on both sides,

values:
  telemetry:
    enabled: true
    v2:
      enabled: true
      stackdriver:
        enabled: true  # This enables Stackdriver metrics
        inboundAccessLogging: FULL  # This enables full Stackdriver server access logging, the other two options are `ERRORS_ONLY` and `NONE`
        outboundAccessLogging: FULL  # This enables Stackdriver full Stackdriver client access logging, the other two options are `ERRORS_ONLY` and `NONE`

Note: values.telemetry.v2.stackdriver.logging option is still supported post 1.7, which could only control enablement of server access log.

You can add the equivalent command line arguments to istioctl command --set values.telemetry.enabled=true --set values.telemetry.v2.enabled=true --set values.telemetry.v2.stackdriver.enabled=true --set values.telemetry.v2.stackdriver.logging=true.

This will apply several EnvoyFilters to your cluster to enabled Metadata exchange filter and Stackdriver filter inside proxy. After enabling, in Stackdriver UI you will find:

• istio.io prefixed metrics be reported
• Server access log named as server-accesslog-stackdriver for every mesh request. Client access log named as client-accesslog-stackdriver Here is an example log entry

{
  "insertId": "1j84zg8g68vb62z",
  "httpRequest": {
    "requestMethod": "GET",
    "requestUrl": "http://35.235.89.201:80/productpage",
    "requestSize": "795",
    "status": 200,
    "responseSize": "7005",
    "remoteIp": "10.168.0.26:0",
    "serverIp": "10.36.3.153:9080",
    "latency": "0.229384205s",
    "protocol": "http"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "cluster_name": "istio-e2e22",
      "namespace_name": "istio-bookinfo-1-68819",
      "container_name": "productpage",
      "project_id": "***",
      "location": "us-west2-a",
      "pod_name": "productpage-v1-64794f5db4-8xbtf"
    }
  },
  "timestamp": "2020-08-13T21:37:42.963881Z",
  "severity": "INFO",
  "labels": {
    "protocol": "http",
    "upstream_host": "127.0.0.1:9080",
    "source_canonical_service": "istio-ingressgateway",
    "source_namespace": "istio-system",
    "x-envoy-original-path": "",
    "source_canonical_revision": "latest",
    "connection_id": "32",
    "upstream_cluster": "inbound|9080|http|productpage.istio-bookinfo-1-68819.svc.cluster.local",
    "requested_server_name": "outbound_.9080_._.productpage.istio-bookinfo-1-68819.svc.cluster.local",
    "destination_version": "v1",
    "destination_workload": "productpage-v1",
    "source_workload": "istio-ingressgateway",
    "destination_canonical_revision": "v1",
    "mesh_uid": "cluster.local",
    "source_principal": "spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account",
    "x-envoy-original-dst-host": "",
    "service_authentication_policy": "MUTUAL_TLS",
    "destination_principal": "spiffe://cluster.local/ns/istio-bookinfo-1-68819/sa/bookinfo-productpage",
    "response_flag": "-",
    "log_sampled": "false",
    "destination_service_host": "productpage.istio-bookinfo-1-68819.svc.cluster.local",
    "destination_name": "productpage-v1-64794f5db4-8xbtf",
    "destination_canonical_service": "productpage",
    "destination_namespace": "istio-bookinfo-1-68819",
    "source_name": "istio-ingressgateway-6845f6d664-lnfvp",
    "source_app": "istio-ingressgateway",
    "destination_app": "productpage",
    "request_id": "39013650-4e62-9be2-9d25-78682dd27ea4",
    "route_name": "default"
  },
  "logName": "projects/***/logs/server-accesslog-stackdriver",
  "trace": "projects/***t/traces/466d77d15753cb4d7749ba5413b5f70f",
  "receiveTimestamp": "2020-08-13T21:37:48.758673203Z",
  "spanId": "633831cb1fda4fd5",
  "traceSampled": true
}

Helm installation does not support these options. If you are using Helm, alternatively you could generate the manifest with istioctl, extract out the necessary EnvoyFilter configuration, and apply them manually. Specifically run istioctl manifest generate with the aforementioned values, look for and apply versioned Stackdriver and Metadata Exchange EnvoyFilter, e.g. stackdriver-filter-1.5, metadata-exchange-1.5.

Non-global Enablement

To enable Stackdriver telemetry for a subset of workloads or a namespace, an EnvoyFilter has to be applied in the namespace, or with desired workload selector. Here is an example EnvoyFilter for 1.7. Please refer to EnvoyFilter and Stackdriver Extension configuration reference for configuration composition.

To get the EnvoyFilter for a particular release as start point for configuration composition, use istioctl to generate the manifest, and get Stackdriver EnvoyFilter (stackdriver-filter-x.x) from the generated manifest.

Trace

To enable tracing, set the following options in istioctl installation: --set values.global.enableTracing=true --set values.global.proxy.tracer=stackdriver. This will enable Envoy proxy to export traces directly to Stackdriver. You can adjust sampling rate using pilot.traceSampling option as other tracers.

Context Graph (Topology)

To enable topology, set the following options in istioctl installation: --set value s.telemetry.v2.stackdriver.topology=true. This will enable Envoy proxy to export context graph edge to Stackdriver.