BoundServiceAccountTokenVolume refresh token

[tomiszili]

Hi!

I builded the latest dockerfile for papertrail, but the latest kubernetes serviceaccount 1hour expiration token can't refreshed by the image.
How can it be solved?

[sthristov]

I believe that the Kubernetes client SDK needs to be updated. I'm trying to update an EKS cluster to 1.22 and this is preventing me from doing it. Here are the AWS docs about BoundServiceAccountTokenVolume

[hugomcfonseca]

👍 on this topic

We are currently running latest version of fluentd-kubernetes-daemonset (fluentd-kubernetes-daemonset:v1.14-debian-kafka2-1) in EKS 1.21, and we are seeing it is using an old token in Cloudwatch logs (using this method).

I believe this is not implemented yet in fluentd-kubernetes-daemonset due to this: ManageIQ/kubeclient#561

[ashie]

I believe this is not implemented yet in fluentd-kubernetes-daemonset due to this: ManageIQ/kubeclient#561

Yes, it's not implemented yet.

I've described the current context at ManageIQ/kubeclient#561 (comment)
Probably we need to implement token refresh feature to kubeclient.

[hugomcfonseca]

@ashie are there any plans for such support?

[ashie]

How about setting /var/run/secrets/kubernetes.io/serviceaccount/token to bearer_token_file of fluent-plugin-kubernetes_metadata_filter?
https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration

As I described at ManageIQ/kubeclient#561, it seems that it's already implemented, but not configured by default (the default value is nil in both kubeclient and fluent-plugin-kubernetes_metadata_filter).

[hugomcfonseca]

@ashie I can give it a try. But I was expecting fluentd to be already using by default as I am not setting any other configuration to ensure fluentd authenticates in Kubernetes API.

[hugomcfonseca]

@ashie I have tried with that setting, although it still appears fluentd to be reusing an old token (probably it does not refresh token as expected?).

For further info, here's the snippet of configuration I added to my fluent.conf:

    <filter kubernetes.var.log.containers.**.log>
      @type kubernetes_metadata
      bearer_token_file /var/run/secrets/kubernetes.io/serviceaccount/token
    </filter>

[ashie]

@ashie I have tried with that setting, although it still appears fluentd to be reusing an old token (probably it does not refresh token as expected?).

Sorry, although it's implemented in master branch of kubeclient, it's not released yet.

[lcohen-11]

@ashie I have tried with that setting, although it still appears fluentd to be reusing an old token (probably it does not refresh token as expected?).

Sorry, although it's implemented in master branch of kubeclient, it's not released yet.

@ashie, Hi,
Do you know when will you released the fix ? thanks in advance.

[ashie]

More refs:

• k8s 1.22 - bearer tokens from mounts needs to be refreshed fabric8io/fluent-plugin-kubernetes_metadata_filter#323
• Use bearer token file and periodically recreate client fabric8io/fluent-plugin-kubernetes_metadata_filter#328
• Refresh k8s client on 'Unathorized' exceptions fabric8io/fluent-plugin-kubernetes_metadata_filter#337

[PettitWesley]

The fix I contributed has been released: fabric8io/fluent-plugin-kubernetes_metadata_filter#323 (comment)

fabric8io/fluent-plugin-kubernetes_metadata_filter#337

https://rubygems.org/gems/fluent-plugin-kubernetes_metadata_filter/versions/2.11.1

EDIT 🚨: Actually this may not work fully. SORRY :( I was still completing final testing and did not think the filter plugin folks would merge and release it until I had stated that my testing is complete. I am working on fixing this. Apologies to everyone.

EDIT 2: I might have panicked too much... I think it does work my testing is just invalid and I need to redo it to fully validate.

[PettitWesley]

@ashie And others, please see my update above.

[ashie]

Thank you for notifying it. I'll wait your report.

[PettitWesley]

@ashie It fully works! Sorry for the alarm!

Testing details are in the PR: fabric8io/fluent-plugin-kubernetes_metadata_filter#337

[ashie]

I see, thanks for your effort!

[ashie]

Now rebuilding images...

[ashie]

Done.

[hugomcfonseca]

@ashie have you tried this?

I used the tag v1.14.6-debian-kafka2-1.1, but still seems to use a token older than 1h after looking for this in cloudwatch following this procedure.

[ashie]

Please see #1368

[hugomcfonseca]

Thank you @ashie. I got it now, it only refreshes it after the 90 days actually.

[PettitWesley]

@ashie @hugomcfonseca Just to confirm since I was the implementor of the workaround, yes, it only refreshes reactively when the token is actually expired. To proactively refresh, a new release of the underlying Kubeclient package is required.

[Ga13Ou]

Did the underlying kubeclient got updated? is there any way to make it refresh each hour to no longer have the stale-token annotation?
If this is not the case, is there a way to test that the fix works without waiting 90 days?