Analyzing Istio Performance

Analyzing Istio Performance

Control Plane

Most control plane components are instrumented with pprof. This allows profiling of memory, CPU usage, etc.

To profile Pilot, follow these steps:

All of these steps assume Pilot is located at localhost:8080. For most situations, this can be achieved with kubectl port-forward -n istio-system PILOT-POD 8080.

More details about pprof can be found here.

Memory

To profile memory usage:

$ go tool pprof -http=:8888 localhost:8080/debug/pprof/heap
Fetching profile over HTTP from http://localhost:8080/debug/pprof/heap
Saved profile in /home/pprof/pprof.pilot-discovery.alloc_objects.alloc_space.inuse_objects.inuse_space.032.pb.gz

Running this will fetch a memory profile from Pilot, open up a web UI, and save the profile to a gz file.

On the web UI (localhost:8888 in this example), you can view current memory and total allocated memory, in different formats. Generally the "Flame Graph" view is the easiest to understand.

CPU

To profile CPU:

$ go tool pprof -http=:8888 localhost:8080/debug/pprof/profile

Note: whereas memory will capture a snapshot of current usage and all lifetime allocations, this will poll for 30s and capture CPU usage during this time.

Goroutines

To debug a goroutine leak or deadlock, it can be useful to dump all active goroutines. This can be done with

curl 'http://localhost:8080/debug/pprof/goroutine?debug=2'

Note: output may be very large

Data Plane

Prerequisite

(For Istio 1.5 and older) In order to write heap profiles, Envoy needs to be able to write to the file system. By default, this is restricted by readOnlyRootFilesystem: true. This can be change in a few ways:

• Install with --set values.global.proxy.enableCoreDump=true
• Manually modify istio-sidecar-injector configmap
• Do a manual injection and remove readOnlyRootFilesystem: true

Profile

On Istio 1.5 and older:

export POD=pod-name
export NS=istio-system
kubectl exec -n "$NS" "$POD" -c istio-proxy -- sh -c 'sudo mkdir -p /var/log/envoy && sudo chmod 777 /var/log/envoy && curl -X POST -s "http://localhost:15000/heapprofiler?enable=y"'
sleep 15
kubectl exec -n "$NS" "$POD" -c istio-proxy -- sh -c 'curl -X POST -s "http://localhost:15000/heapprofiler?enable=n"'
rm -rf /tmp/envoy
kubectl cp -n "$NS" "$POD":/var/log/envoy/ /tmp/envoy -c istio-proxy
kubectl cp -n "$NS" "$POD":/lib/x86_64-linux-gnu /tmp/envoy/lib -c istio-proxy
kubectl cp -n "$NS" "$POD":/usr/local/bin/envoy /tmp/envoy/lib/envoy -c istio-proxy

On Istio 1.6+

export POD=pod-name
export NS=istio-system
export PROFILER="cpu" # Can also be "heap", for a heap profile
kubectl exec -n "$NS" "$POD" -c istio-proxy -- curl -X POST -s "http://localhost:15000/${PROFILER}profiler?enable=y"
sleep 15
kubectl exec -n "$NS" "$POD" -c istio-proxy -- curl -X POST -s "http://localhost:15000/${PROFILER}profiler?enable=n"
rm -rf /tmp/envoy
kubectl cp -n "$NS" "$POD":/var/lib/istio/data /tmp/envoy -c istio-proxy
kubectl cp -n "$NS" "$POD":/lib/x86_64-linux-gnu /tmp/envoy/lib -c istio-proxy
kubectl cp -n "$NS" "$POD":/usr/local/bin/envoy /tmp/envoy/lib/envoy -c istio-proxy

Visualize profile pprof installation

Install pprof, then run:

PPROF_BINARY_PATH=/tmp/envoy/lib/ pprof -pdf /tmp/envoy/lib/envoy /tmp/envoy/envoy.prof

Or, interactively

PPROF_BINARY_PATH=/tmp/envoy/lib/ pprof /tmp/envoy/lib/envoy /tmp/envoy/envoy.prof

Or, through the web UI

PPROF_BINARY_PATH=/tmp/envoy/lib/ pprof -http=localhost:8000 /tmp/envoy/lib/envoy /tmp/envoy/envoy.prof