Istioctl troubleshooting without control plane access for External Istiod

How to get istioctl version (and friends) working under External Istiod.

Istio 1.8 will not expose the older insecure debug endpoints that depended on K8s port forwarding for security.

If the Istio control plane runs in your own cluster nothing changes.

Exposing an external control plane for istioctl troubleshooting

If your control plane is hosted outside your mesh, and you do not have Kubernetes API access to it, Istiod must be securely exposed.

• Developers' Note: If you are developing just use kubectl -n istio-system port-forward deployment/istiod 15012 for testing.

We will use Istio to expose Istio itself via a gateway.

The script below exposes the Istiod port 15012 to the outside world. Anyone attempting to contact will need certificates or Kubernetes tokens to contact Istiod XDS. I will show the client set up in the next step.

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istiod-xds-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15443
      name: tcp
      protocol: HTTPS
    tls:
      mode: PASSTHROUGH
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: istiod-xds
  namespace: istio-system
spec:
  hosts:
  - "*"
  gateways:
  - istiod-xds-gateway
  tls:
  - match:
    - port: 15443
      sniHosts:
      - istiod.istio-system.svc
    route:
    - destination:
        host: istiod.istio-system.svc.cluster.local
        port:
          number: 15012
EOF

At this point, the outside world can contact Istiod, but TLS is used.

Get your exposed Istio XDS endpoint address:

INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
ISTIOD_ENDPOINT=$INGRESS_HOST:15433
echo External Istiod endpoint is $ISTIOD_ENDPOINT

This address is the one to give users.

We must now configure a client to use one of the two security methods.

Token method

As of 1.8-dev istioctl, no certificates are needed.

Use istioctl experimental proxy-status --xds-address istio.cloudprovider.example.com:15012 to get the proxy-status from external Istiod. (If your Istio control plane is not external you may leave off the --xds-address flag.)

That's all. There is no need to create certificates. istioctl does need access to a Kubernetes cluster managed by an external Istiod control plane.

Make your external Istiod endpoint the default

Create an istioctl configuration file. Istio configuration files are a new, dark-launched feature of Istio 1.7.

mkdir -p ~/.istioctl
echo "discoveryAddress: $ISTIOD_ENDPOINT" > ~/.istioctl/config.yaml

At this point, the experimental XDS-based External Istiod commands can be used without flags, like this: istioctl x version.

YOU DO NOT NEED TO CONTINUE FUTHER WITH THESE INSTRUCTIONS.

Non-Kubernetes method

Certificates can be used to access the control plane without Kubernetes. This method is not recommended, but is documented here to show how Istio can be used without Kubernetes.

The connection to external Istiod is secured. The process to get certificates is not part of istioctl. You need a dev environment and access to the cluster where external Istiod is running to create certificates and expose XDS.

First, create certificates for the control plane cluster

This step requires admin access to the cluster running external Istiod.

Set ISTIO_DIR to your Istio source code, e.g. ISTIO_DIR=~/go/src/istio.io/istio.

cd $ISTIO_DIR/tools/certs
NICKNAME=itp # Use your control plane cluster name, or a private nickname
make istioctl-${NICKNAME}-cacerts-k8s
make istioctl-${NICKNAME}-certs-k8s
rm istioctl-${NICKNAME}/cert-chain.pem
cat istioctl-${NICKNAME}/workload-cert.pem istioctl-${NICKNAME}/workload-cert-chain.pem > istioctl-${NICKNAME}/cert-chain.pem

At this point you have certificates in your istioctl-${NICKNAME} directory.

For this tutorial we put the certs in /tmp. You may want to put them into your $HOME directory.

Secure external Istiod endpoint

• Developers' Note: Use ISTIOD_ENDPOINT=localhost:15012 for testing with kubectl port-forward or local pilot-discovery testing

At this point you no longer require Kubernetes access to the Istio control plane. Everything from here down should work without $KUBECONFIG or ~/.kube/config.

Even logged out of Kubernetes, the experimental version and ps commands will work if you supply flags for the location and security of Istiod XDS. Using the $ISTIO_ENDPOINT defined above,

istioctl x version --xds-address $ISTIOD_ENDPOINT --authority istiod.istio-system.svc --cert-dir istioctl-${NICKNAME}

Make these settings the default

Create an istioctl configuration file. Istio configuration files are a new, dark-launched feature of Istio 1.7.

mkdir -p ~/.istioctl
echo "discoveryAddress: $ISTIOD_ENDPOINT" > ~/.istioctl/config.yaml
echo "authority: istiod.istio-system.svc" >> ~/.istioctl/config.yaml
echo "cert-dir: $(pwd)/istioctl-$NICKNAME" >> ~/.istioctl/config.yaml

At this point, the experimental XDS-based external Istiod compatible implementations can be used without flags, like this: istioctl x version.

• Developers' Note: To see the other istioctl configurable parameters, use istioctl x config list.

Replace the regular version and ps commands with the experimental XDS-based implementations

Turn on an experimental dark-launched feature to replace the cluster-based implementations with the experimental implementations that support external Istiod.

echo "prefer-experimental: true" >> ~/.istioctl/config.yaml

Once prefer-experimental is turned on, all of the experimental XDS implementation commands replace legacy commands.

In Istio 1.8 we intend to replace the legacy versions with the experimental versions.

Cleanup

Log back into Kubernetes.

Adminstrators can do kubectl -n istio-system delete vs istiod-xds and kubectl -n istio-system delete gw istiod-xds-gateway to un-expose their Istio XDS.

Users can do rm ~/.istioctl/config.yaml to remove istioctl configuration defaults.