Getting istio proxy coredumps

When istio-proxy crashes you can see the restart count go up for the pod. The proxy logs it will say that it crashed with some back trace information.

1. Enable core dumps

   Set global.proxy.enableCoreDump=true

./istioctl-1.5.2 manifest apply --set values.global.proxy.enableCoreDump=true

Note: Additional Steps prior to 1.5.3

kubectl -n istio-system edit deployment istio-ingressgateway

Update the first initContainer and update the command line to send coredumps to an alternate location. Look for a command start with sysctl -w and change the kernel.core_pattern=/var/run/ingress_gateway/core.proxy

Note: Prior to Istio 1.5.3, there is a bug that makes it impossible collect the coredump, since the container dies. The fix is to write coredump in an emptyDir or to add a pre-stop hook. This has been fixed in 1.5.3 onwards.

1. Find the proxy used. Look for PROXY_REPO_SHA section and the lastStableSHA key in it. This represents the proxy sha used in istio-proxy build.

   https://github.com/istio/istio/blob/${istio_version}/istio.deps#L7

   To find proxy_sha for version 1.5.2 for example use istio_version=1.5.2

   https://github.com/istio/istio/blob/1.5.2/istio.deps#L7

   latestStableSHA=ff8d26a507d4e75863c9800c143a5b94b2a3fdd5

2. Download proxy the image with debug symbols in a local ${WORKDIR}

   wget https://storage.googleapis.com/istio-build/proxy/envoy-symbol-${latestStableSHA}.tar.gz

   In the example above it will be

    export istio_version=1.5.2
    export latestStableSHA=ff8d26a507d4e75863c9800c143a5b94b2a3fdd5
    export WORKDIR=...
    cd ${WORKDIR}
    wget https://storage.googleapis.com/istio-build/proxy/envoy-symbol-${latestStableSHA}.tar.gz
    tar -xzvf envoy-symbol-${latestStableSHA}.tar.gz

   example: https://storage.googleapis.com/istio-build/proxy/envoy-symbol-ff8d26a507d4e75863c9800c143a5b94b2a3fdd5.tar.gz

3. Download coredumps from the proxy in question in ${WORKDIR}

    $ docker run -it -v ${WORKDIR}:/work --entrypoint=/bin/bash istio/proxyv2:${istio_version}
    $ apt-get update && apt-get install -y gdb
    $ gdb /work/usr/local/bin/envoy /work/proxy.core

You can run bt in gdb to get full backtrace.