Unable to use the mutation webhook method with kubernetes 1.21.5 and admissionregistration.k8s.io/v1 on GKE

[martinlevesque]

Hi everyone,

We have been using the following way to use berglas with mutation webhook https://github.com/GoogleCloudPlatform/berglas/tree/main/examples/kubernetes with admissionregistration.k8s.io/v1beta1 mutation webhook with success. One important note is that admissionregistration.k8s.io/v1beta1 will get deprecated in Kubernetes 1.22 in favor of admissionregistration.k8s.io/v1.

As soon as we try to use admissionregistration.k8s.io/v1 we are no more able to have it work using the go function here https://github.com/GoogleCloudPlatform/berglas/tree/main/examples/kubernetes

From the gcloud function everything looks working, as we see logs such as:

2022/01/11 19:29:41 [DEBUG] json patch for request 82e6db3e-7e4e-499c-ae83-dcd622ce168b: [{"op":"add","path":"/spec/volumes/1","value":{"name":"berglas-bin","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/initContainers","value":[{"name":"copy-berglas-bin","image":"us-docker.pkg.dev/berglas/berglas/berglas:latest","command":["sh","-c","cp /bin/berglas /berglas/bin/"],"resources":{},"volumeMounts":[{"name":"berglas-bin","mountPath":"/berglas/bin/"}],"imagePullPolicy":"IfNotPresent"}]},{"op":"replace","path":"/spec/containers/0/command/0","value":"/berglas/bin/berglas"},{"op":"add","path":"/spec/containers/0/args","value":["exec","--","/bin/envserver"]},{"op":"add","path":"/spec/containers/0/volumeMounts/1","value":{"name":"berglas-bin","readOnly":true,"mountPath":"/berglas/bin/"}}]

However the POD does not start and I can see the following POD status error:

- lastTransitionTime: "2022-01-12T16:25:42Z"
    lastUpdateTime: "2022-01-12T16:25:42Z"
    message: 'Internal error occurred: failed calling webhook "berglas-webhook.cloud.google.com":
      expected webhook response of admission.k8s.io/v1, Kind=AdmissionReview, got
      /, Kind='
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure

The webhook file I am using:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: berglas-webhook
  labels:
    app: berglas-webhook
    kind: mutator

webhooks:
- name: berglas-webhook.cloud.google.com
  admissionReviewVersions: ["v1", "v1beta1"]
  clientConfig:
    url: https://northamerica-northeast1-PROJECT-ID.cloudfunctions.net/berglas-secrets-webhook
    caBundle: ""
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  sideEffects: NoneOnDryRun
  

I also tried to update 2 packages (berglas and kubewebhook) in go.mod but I am getting the same behavior:

module hello

go 1.13

require (
    github.com/GoogleCloudPlatform/berglas v0.6.2
    github.com/slok/kubewebhook/v2 v2.1.0
    k8s.io/api v0.18.6
    k8s.io/apimachinery v0.18.6
)

Any ideas what should be changed to make it work with admissionregistration.k8s.io/v1 ?

Thanks!

• Related issue reporting the same issue: kubewebhook is outdated #162

[martinlevesque]

ping! No auto close please :-)

[peterldowns]

Also interested in the response here, I have never worked on kubernetes things before but we would like to upgrade our cluster to 1.22 (currently on 1.21) and the MutatingWebhookConfiguration beta api "will no longer be served". Happy users of Berglas, but not sure how to make this fix ourselves.

[sethvargo]

Some k8s tools upgraded to go 1.17 without a regard for backwards compat, which is making this impossible to deploy to Cloud Functions right now, since the latest available version there is 1.16.

• install controller-gen failed: sf.IsExported undefined kubernetes-sigs/controller-tools#643
• Project support for Go 1.16 kubernetes-sigs/json#8

#180 is the update PR. You can put it in a container and run it on Cloud Run, but there's no way to run it on Cloud Functions right now.

[peterldowns]

@sethvargo thank you for the explanation, and for having already fixed this and gotten it ready to go in your PR! I see that you are a member of Google Cloud Platform and work at Google -- is there anything I can do to help request that Cloud Functions add a go1.17/go1.18 runtime? I don't know if you're directly involved, but maybe there's a better place to ask than here? Thank you again for your time and for maintaining Berglas, it's been a wonderfully simple solution that's let us use the Google Cloud Secrets Manager for everything both inside k8s and outside.

[sethvargo]

I don't have any control over that, sorry.

[peterldowns]

No problem. We'll just deploy the Cloud Run container instead. Thank you again for your support.

[martinlevesque]

@peterldowns

if I understand correctly, the changes in #180 can be used with Cloud Run, as the Dockerfile/build can use the latest go (1.17) version?
Will try it also in few weeks/months also.

[sethvargo]

Correct - you can use Cloud Run. I didn't want to switch to Cloud Run because it introduces complexity that folks who aren't familiar with Docker or containers might not want to take on.

[martinlevesque]

After looking at the gcloud run samples, my current solution for "typical" kubernetes deployments without using the kubernetes webhook approach:

• Add this line in the Dockerfile to add the berglas binary: COPY --from=us-docker.pkg.dev/berglas/berglas/berglas:latest /bin/berglas /bin/berglas
• In the deployment container spec, set a command line (example with python): command: ["/bin/berglas", "exec", "--", "python", "server.py"]
• Ensure to have a proper kubernetes service account in the namespace.

The remaining thing is to make it work for helm charts, which do not have the /bin/berglas binary. Many helms allow to pass a command, so this part is fine. For the /bin/berglas binary, I guess you could inject it using an initContainer and mount a volume if the helm has initContainer parameters. But if it does not have initContainer parameters, not sure how it can be done. Any ideas?

[sethvargo]

Right - you can do this without the webhook. The webhook just automates the steps you described above (although it uses a sidecar instead).

[github-actions]

This issue is stale because it has been open for 14 days with no
activity. It will automatically close after 7 more days of inactivity.