Envoy native telemetry

It is possible to get a subset of the full istio telemetry thru envoy only metrics. Note: by Istio 1.3/1.4 the proxy will emit full istio telemetry.

Envoy native telemetry has the following feature gaps compared to Istio standard metrics.

• Configurability
• mTLS indicator
• Peer service account
• Source information

How to enable Envoy native telemetry

Istio 1.2 supports stats annotations to specify metrics to be collected by Envoy.

Annotation	Description
sidecar.istio.io/statsInclusionPrefixes	Specifies the comma separated list of prefixes of the stats to be emitted by Envoy.
sidecar.istio.io/statsInclusionRegexps	Specifies the comma separated list of regexes the stats should match to be emitted by Envoy.
sidecar.istio.io/statsInclusionSuffixes	Specifies the comma separated list of suffixes of the stats to be emitted by Envoy.

Ingress gateway

Envoy native metrics provide good client side metrics. The source is fully specified by the prometheus scraper and the destination information is encoded in the upstream cluster name.

The following stats set at ingress gateway gives a useful set of metrics. You can compute clientside traffic-by-responsecode and associated latency histograms. It also gives you downstream metrics by port. Since ingress gateway does not use the sidecar injector, you have to configure it as follows using environment variables.

      containers:
      - name: istio-proxy
        env:
        - name: ISTIO_METAJSON_STATS
          value: |
            {"sidecar.istio.io/statsInclusionSuffixes": "upstream_rq_1xx,upstream_rq_2xx,upstream_rq_3xx,upstream_rq_4xx,upstream_rq_5xx,upstream_rq_time,upstream_cx_tx_bytes_total,upstream_cx_rx_bytes_total,upstream_cx_total,downstream_rq_1xx,downstream_rq_2xx,downstream_rq_3xx,downstream_rq_4xx,downstream_rq_5xx,downstream_rq_time,downstream_cx_tx_bytes_total,downstream_cx_rx_bytes_total,downstream_cx_total"}

You can import a sample dashboard based on the above configuration, it should look like this

Sidecar

For a sidecar we collect the full set of upstream metrics. However, only remote downstream metrics are usually of interest. You can configure them as follows thru an annotation.

    "sidecar.istio.io/statsInclusionSuffixes": "upstream_rq_1xx,upstream_rq_2xx,upstream_rq_3xx,upstream_rq_4xx,upstream_rq_5xx,upstream_rq_time,upstream_cx_tx_bytes_total,upstream_cx_rx_bytes_total,upstream_cx_total"
    "sidecar.istio.io/statsInclusionPrefixes": "http.{pod_ip}_"

The {pod_ip} is automatically substituted during bootstrap config generation.

1.1

1.1 does not automatically add required stats and does not support suffixes.

Add the following annotation to a deployment where you want to collect metrics.

"sidecar.istio.io/statsInclusionPrefixes": "http.,tcp.,cluster.,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grpc,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grpc"

For gateways add the equivalent env variables as shown in a previous section.