Configurable V2 Metrics

Configurable Metrics without Mixer

Istio 1.5 includes an experimental support for modifying the standard metrics.

The API to configure the metrics is certain to change in Istio 1.6 as part of the extensions API design.

To enable this feature, two steps must be performed:

1. Configure EnvoyFilter to add, remove, or edit the dimensions and the metrics produced by Istio stats extension.

2. Annotate pods to extract the extra dimensions to the prometheus stats.

Step 1: Configuring EnvoyFilter

Locate the envoy.wasm.stats extension configuration. The default configuration is in the configuration section and typically looks like this:

{
  "debug": "false",
  "stat_prefix": "istio"
}

Augment the configuration section for each instance of the extension configuration. For example, to add destination_port and request_host dimensions to the standard requests_total metric, change it to look like this:

{
   "debug": "false",
   "stat_prefix": "istio",
   "metrics": [
      {
        "name": "requests_total",
        "dimensions": {
           "destination_port": "destination.port",
           "request_host": "request.host"
        }
      }
   ]
}

Note that all the metric names are automatically prefixed with istio_, so you need to omit the prefix from the name field in the metric specification.

Stats extension configuration overview

The configuration consists of two parts: definitions and metrics. The definition section supports creating entirely new metrics by name, the expected value expression, and the metric type (counter, gauge, histogram). The metric section provides values for the dimensions of the metric as expressions, and also allows removing or overriding the existing metric dimensions.

By default, Istio defines a set of the standard metrics (e.g. requests_total). The configuration is resolved relative to the standard metrics, e.g. with no configuration the standard metrics are emitted. However, using tags_to_remove or by re-defining a dimension, the standard metrics can be modified.

See the reference specification.

Expressions used for values

The values in the metric configuration are common expressions. That means, for example, that strings must be double-quoted in JSON, e.g. "'some_string_value'". Unlike Mixer expression language, there is no support for the pipe (|) operator at the moment, but it can be emulated with either has or in: has(request.host) ? request.host : "unknown".

All the standard Envoy attributes are exposed. Additionally, a few extra attributes can be used:

Attribute	Type	Value
listener_direction	int64	Enum value for listener direction
listener_metadata	metadata	Per-listener metadata
route_metadata	metadata	Per-route metadata
cluster_metadata	metadata	Per-cluster metadata
node	node	Node description
cluster_name	string	Upstream cluster name
route_name	string	Route name
filter_state ** (1.5 only)	map<string, bytes>	Per-filter state blob, any name not covered elsewhere is searched in the filter_state starting 1.6. So specifying filter_state['key'] is not supported.
plugin_name	string	Wasm extension name
plugin_root_id	string	Wasm root instance ID
plugin_vm_id	string	Wasm VM ID

Step 2: Annotating pods

Each injected pod should have an annotation with the list of the dimensions to be extracted into Prometheus time series. The following annotation should be applied to all injected pods for the example above:

apiVersion: extensions/v1beta1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/extraStatTags: destination_port,request_host