Context aware policies are not working with rusttls

[flavio]

Now that we switched to rusttls, the thread fetching information from a Kubernetes cluster is not able to connect to the cluster anymore.

My current setup:

• policy-server running from main, with rusttls enabled
• external kuberenetes cluster: built using k3s

This is the error message we get:

WARN policy_server::kube_poller: error when initializing the cluster context client: could not initialize a cluster context because a Kubernetes client could not be created: SslError: No valid private key was found

[viccuad]

After debugging, it seems we are hitting the following issues in the kube crate, when feature = {rustls}:

• rustls cannot reach a cluster through ip kube-rs/kube#153
• rustls cannot use client-key in EC Private Keys format kube-rs/kube#542

First, with kube = 0.64.0, and k3d, one is hitting kube-rs/kube#153 (kube client with rustls can't reach cluster through ip. Workaround: edit kubeconfig and use localhost). A fix is partially fixed in kube since 0.59.0, but blocked on rustls issues.

After applying workarund, and bumping kube to 0.65.0 (which needs a bump in policy-evaluator from kube = 0.64.0 to 0.65.0), I get a more descriptive error: identity PEM is missing a private key: the key must be PKCS8 or RSA/PKCS1. This corresponds with the second issue. Workaround, as listed in the issue, is to convert the key in the kubeconfig to PKCS8. In my case:

kubectl config view --raw \
  -o jsonpath='{.users[?(@.name == "admin@k3d-k3s-default")].user.client-key-data}' | \
  base64 -d | openssl pkcs8 -topk8 -nocrypt | base64 -w0 | \
  xargs -I{} kubectl config set users.admin@k3d-k3s-default.client-key-data {}

With minikube and kube 0.65.0, I hit the dns issue (as for me it ends for example in https://192.168.49.2:8443). A workaround is to add an entry to /etc/hosts.

[jvanz]

AFAICS in my test, upgrading the kube crate to v0.70.0 as suggested during a call with the Kubewarden team, solve the identity PEM is missing a private key: the key must be PKCS8 or RSA/PKCS1 issue. So, I propose upgrade the lib and document how to workaround the ip issue.

I'll open the PRs.

[jvanz]

The related PR released in the kube crate version 0.70.0: kube-rs/kube#804