Unauthorized error with exec auth with eks

[aviramha]

Current and expected behavior

kubectl get pods of the go client works
kubectl get pods of the kube-rs example fails with "Unauthorized error" (see trace log below)
We stumbled upon this bug in metalbear-co/mirrord#984 and thought maybe it's a kube-rs issue, we reproduced it with the example so we believe it's not bad usage causing it.

Possible solution

No response

Additional context

Relevant kubeconfig

apiVersion: v1
clusters:
- cluster:
  certificate-authority-data: <redacted>
  server: http://127.0.0.1:6443  name: kubernetescontexts:- context:
  cluster: kubernetes
  namespace: <redacted>
  user: <redacted>
  name: aws
  current-context: aws
  kind: Config
  preferences: {}
  users:
    - name: aws
      user:
        exec:
          apiVersion: client.authentication.k8s.io/v1alpha1
          args:
          - eks
          - get-token
          - --cluster-name
          - <redacted>
          command: aws
          env: null
          provideClusterInfo: false

RUST_LOG=trace ./kubectl get pods
2023-01-25T15:00:49.793403Z TRACE tower::buffer::service: sending request to buffer worker
2023-01-25T15:00:49.794407Z TRACE tower::buffer::worker: worker polling for next message
2023-01-25T15:00:49.794439Z TRACE tower::buffer::worker: processing new request
2023-01-25T15:00:49.794471Z TRACE tower::buffer::worker: resumed=false worker received request; waiting for service readiness
2023-01-25T15:00:49.794495Z DEBUG tower::buffer::worker: service.ready=true processing request
2023-01-25T15:00:49.794541Z TRACE tower::buffer::worker: returning response future
2023-01-25T15:00:49.794580Z TRACE tower::buffer::worker: worker polling for next message
2023-01-25T15:00:49.794644Z DEBUG HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: kube_client::client::builder: requesting
2023-01-25T15:00:49.794683Z TRACE HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: hyper::client::pool: checkout waiting for idle connection: ("http", 127.0.0.1:6443)
2023-01-25T15:00:49.794795Z TRACE HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: hyper::client::connect::http: Http::connect; scheme=Some("http"), host=Some("127.0.0.1"), port=Some(Port(6443))2023-01-25T15:00:49.797079Z DEBUG HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: hyper::client::connect::http: connecting to 127.0.0.1:6443
2023-01-25T15:00:49.797312Z TRACE HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: mio::poll: registering event source with poller: token=Token(0), interests=READABLE | WRITABLE
2023-01-25T15:00:49.797383Z DEBUG HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: hyper::client::connect::http: connected to 127.0.0.1:6443
2023-01-25T15:00:49.797411Z TRACE HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: hyper::client::conn: client handshake Http1
2023-01-25T15:00:49.797444Z TRACE HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: hyper::client::client: handshake complete, spawning background dispatcher task
2023-01-25T15:00:49.797549Z TRACE want: signal: Want
2023-01-25T15:00:49.797572Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
2023-01-25T15:00:49.798975Z TRACE HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: want: poll_want: taker wants!
2023-01-25T15:00:49.799030Z TRACE HTTP{http.method=GET http.url=http://127.0.0.1:6443/apis otel.name="HTTP" otel.kind="client"}: hyper::client::pool: checkout dropped for ("http", 127.0.0.1:6443)
2023-01-25T15:00:49.799131Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=GET, body=None
2023-01-25T15:00:49.799180Z DEBUG hyper::proto::h1::io: flushed 561 bytes
2023-01-25T15:00:49.799200Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2023-01-25T15:00:49.951682Z TRACE hyper::proto::h1::conn: Conn::read_head
2023-01-25T15:00:49.951723Z TRACE hyper::proto::h1::io: received 330 bytes
2023-01-25T15:00:49.951762Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=330
2023-01-25T15:00:49.951791Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(201)
2023-01-25T15:00:49.951821Z DEBUG hyper::proto::h1::io: parsed 5 headers
2023-01-25T15:00:49.951840Z DEBUG hyper::proto::h1::conn: incoming body is content-length (129 bytes)
2023-01-25T15:00:49.951890Z TRACE hyper::proto::h1::decode: decode; state=Length(129)
2023-01-25T15:00:49.951909Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-01-25T15:00:49.951920Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2023-01-25T15:00:49.951944Z TRACE want: signal: Want
2023-01-25T15:00:49.951964Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2023-01-25T15:00:49.951973Z TRACE want: signal: Want
2023-01-25T15:00:49.951978Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2023-01-25T15:00:49.951970Z TRACE want: poll_want: taker wants!
2023-01-25T15:00:49.952007Z TRACE hyper::client::pool: put; add idle connection for ("http", 127.0.0.1:6443)
2023-01-25T15:00:49.952026Z DEBUG hyper::client::pool: pooling idle connection for ("http", 127.0.0.1:6443)
2023-01-25T15:00:49.952114Z DEBUG kube_client::client: Unsuccessful: ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }
2023-01-25T15:00:49.952138Z TRACE want: signal: Want
2023-01-25T15:00:49.952161Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2023-01-25T15:00:49.952170Z TRACE tower::buffer::worker: buffer already closed
2023-01-25T15:00:49.952192Z TRACE mio::poll: deregistering event source from poller
2023-01-25T15:00:49.952265Z TRACE want: signal: Closed
Error: ApiError: Unauthorized: Unauthorized (ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }) Caused by:
    Unauthorized: Unauthorized

Environment

kubectl version --short

kubectl version --short
Client Version: v1.22.1
Server Version: v1.20.15-eks-fb459a0

Configuration and features

cargo build --release --example kubectl --features rustls-tls,k8s-openapi/v1_26,runtime --no-default-features

Affected crates

kube-core, kube-client, kube-runtime

Would you like to work on fixing this bug?

yes

[goenning]

I've used the aws auth plugin quite a bit and a lot of my clients uses it as well via kube-rs.

You have redacted this part user: <redacted> but this info is not sensitive. Is the actual value user: aws? One thing you could try is adding a few logs around the functions that use the CLI to generate the token, just to confirm the token is being generated and it is correct

[aviramha]

@goenning
Thanks for the help! The issue is not on my end so I'll try to talk to the user who had it to have better understanding of what happens.
I have a loose theory that since they use some sort of http proxy without the http proxy feature, kube rs doesn't use SSL then doesn't use client certificate which leads to unauthorized response... not sure why the go codebase is different though..

[aviramha]

I investigated it further - the setup they use is they have a remote machine running kubectl proxy then they point the local kubeconfig to that remote machine via SSH port forwarding.
This means that the remote kubectl should handle authentication, so I guess that what happens is that kube-rs adds authentication headers in local that aren't overridden in remote, unlike Go's kubectl which doesn't add - not sure why though?

[aviramha]

Debugged it further - we found out that Go's kubectl doesn't send Authorization header and completely disregards the user/auth stuff when on http as a defensive mechanism - kubernetes/kubectl#744
I'll try to send a PR that aligns the logic.
@clux wdyt?

[clux]

Ah, yeah, that sounds like a good plan on paper. Without having looked too deeply, it sounds like we can probably change the Auth type in that case when proxying. A PR will definitely be appreciated.

[aviramha]

Great! either I or someone from our team will send a PR. It might take few weeks though but will let you know if something changes. You can assign it to me meanwhile :)

[bryantbiggs]

not sure if its related but you should update your local version of the awscli so that when you run aws eks update-kubeconfig --name ..., it will pull the correct auth API version. Right now you are getting apiVersion: client.authentication.k8s.io/v1alpha1 and it should be apiVersion: client.authentication.k8s.io/v1beta1

aws/aws-cli#6920