Understanding IPTables snapshot
From time to time, we need to find out the iptables that are active inside a proxy container.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
adservice-fcb677f45-nrmms 2/2 Running 0 176m
cartservice-69958cd8f6-bf9k6 2/2 Running 0 176m
checkoutservice-5c5f9d4887-mmgg4 2/2 Running 0 176m
currencyservice-6dc7f65785-sqxd9 2/2 Running 0 176m
emailservice-5cbcc9d99-9wvlc 2/2 Running 0 176m
Consider that we want to get iptables for the adservice-fcb677f45-nrmms
$ NODENAME=$(kubectl get pods adservice-fcb677f45-nrmms -o jsonpath="{.spec.nodeName}")
$ echo $NODENAME
gke-prod-us-west2-wi-pool-11f4e4b5-2pl6Find the zone where this is located.
$ gcloud compute instances list --filter="name=${NODENAME}" --project "${PROJECT}"
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
gke-prod-us-west2-wi-pool-11f4e4b5-2pl6 us-west2-a n1-standard-4 10.2.0.34 35.236.18.127 RUNNING
zone is us-west2-a
gcloud beta compute ssh --zone us-west2-a "${NODENAME}" --project ${PROJECT}
If running on Container optimized images(cos), start toolbox
Find pod ip
$ kubectl get pods adservice-fcb677f45-nrmms -o jsonpath="{.status.podIP}"
172.16.0.28
On the node use podip to find the pid of the istio-proxy.
The istio-proxy commandline uses the sidecar~${pod_ip}... in the example below it is sidecar~172.16.0.28~adservice-fcb677f45...
[ssh node]$ sudo ps auxf | grep 172.16.0.28
1337 987449 0.5 0.3 206120 60064 ? Sl May28 1:12 \_ /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster adservice.hipster-shop --service-node sidecar~172.16.0.28~adservice-fcb677f45-nrmms.hipster-shop~hipster-shop.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error --concurrency 2
We use nsenter to get iptables.
[ssh node]$ sudo nsenter -t ${PROXY_PID} -n iptables-save
# Generated by iptables-save v1.6.1 on Fri May 29 05:00:49 2020
*nat
:PREROUTING ACCEPT [12921:775260]
:INPUT ACCEPT [12924:775440]
:OUTPUT ACCEPT [9987:716510]
:POSTROUTING ACCEPT [10019:718430]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
- 1337 - uid and gid used to distinguish between traffic originating from proxy vs the applications.
- 127.0.0.6 -- InboundPassthroughBindIpv4. When Istio-proxy passes through an inbound connection to the local workload using the original destination IP (
ORIGINAL_DST), it uses this source ip address. Iptables rules use this source ip to avoid redirection back to the proxy on the output chain.
[ssh node]$ sudo nsenter -t 987449 -n iptables -L -t nat -v
Chain PREROUTING (policy ACCEPT 13403 packets, 804K bytes)
pkts bytes target prot opt in out source destination
13406 804K ISTIO_INBOUND tcp -- any any anywhere anywhere
Chain INPUT (policy ACCEPT 13406 packets, 804K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 10483 packets, 752K bytes)
pkts bytes target prot opt in out source destination
6817 409K ISTIO_OUTPUT tcp -- any any anywhere anywhere
Chain POSTROUTING (policy ACCEPT 10516 packets, 754K bytes)
pkts bytes target prot opt in out source destination
Chain ISTIO_INBOUND (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN tcp -- any any anywhere anywhere tcp dpt:ssh
0 0 RETURN tcp -- any any anywhere anywhere tcp dpt:15090
13403 804K RETURN tcp -- any any anywhere anywhere tcp dpt:15020
3 180 ISTIO_IN_REDIRECT tcp -- any any anywhere anywhere
Chain ISTIO_IN_REDIRECT (3 references)
pkts bytes target prot opt in out source destination
3 180 REDIRECT tcp -- any any anywhere anywhere redir ports 15006
Chain ISTIO_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- any lo 127.0.0.6 anywhere
0 0 ISTIO_IN_REDIRECT all -- any lo anywhere !localhost owner UID match 1337
3572 214K RETURN all -- any lo anywhere anywhere ! owner UID match 1337
3212 193K RETURN all -- any any anywhere anywhere owner UID match 1337
0 0 ISTIO_IN_REDIRECT all -- any lo anywhere !localhost owner GID match 1337
0 0 RETURN all -- any lo anywhere anywhere ! owner GID match 1337
0 0 RETURN all -- any any anywhere anywhere owner GID match 1337
0 0 RETURN all -- any any anywhere localhost
33 1980 ISTIO_REDIRECT all -- any any anywhere anywhere
Chain ISTIO_REDIRECT (1 references)
pkts bytes target prot opt in out source destination
33 1980 REDIRECT tcp -- any any anywhere anywhere redir ports 15001
If you observe the iptables rules above, you will see a magic number (1337) appear several times. This is the uid/gid of the running proxy which is used by iptables to differentiate between packets originating from the proxy and the ones originating from the application. When packets originate from the application, they must be redirected to proxy to implement a service mesh. When packets originate from the proxy, they must not be redirected as doing so will cause an infinite loop.
The effect of the above magic number is that an application running as 1337 uid/gid is not captured and therefore bypasses the proxy. This part can be problematic because this behaviour can happen inadvertently, or by deliberately setting the uid/gid.
Istio also allows explicit proxy bypass on a per workload basis. This can be done by opting out of sidecar injection, or by using traffic.sidecar.istio.io/excludeOutboundIPRanges and traffic.sidecar.istio.io/excludeOutboundPorts annotations to affect the corresponding exclusions.
Outbound bypass can cause multiple issues.
- Outbound traffic is broken in a strict mTLS cluster since the proxy does not get a chance to mediate the traffic.
- If the cluster is not mTLS, the outbound policies could be bypassed such as
Data leak prevention
Until these issue are fixed, a user should not rely on outbound policies.
