Pod is still using old tokens after upgrade

[oganekoi]

Hi,

I am trying this: #1367
However, the token does not seem to be refreshed.
I am using fluentd-kubernetes-daemonset:v1.14.6-debian-cloudwatch-1.1 in AWS EKS 1.22 and I have checked, if it is using stale tokens.

When the API server receives requests with tokens that are older than one hour, then it annotates the pod with annotations.authentication.k8s.io/stale-token. In my case I can see the following annotation. E.g.:

annotations.authentication.k8s.io/stale-token subject: system:serviceaccount:amazon-cloudwatch:fluentd, seconds after warning threshold: 14655

I have also confirmed in the pod logs that the version of fluent-plugin-kubernetes_metadata_filter is 2.11.1.

Pod Logs:

2022-06-30 03:21:52 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.11.1'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-06-30 03:21:52 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2022-06-30 03:21:52 +0000 [info]: gem 'fluentd' version '1.14.6'

Fluentd Kubernetes Daemonset Version Info

fluentd-kubernetes-daemonset:v1.14.6-debian-cloudwatch-1.1

Cluster Details

AWS EKS 1.22
fluentd-kubernetes-daemonset:v1.14.6-debian-cloudwatch-1.1 deployed as Daemonset

Steps to reproduce issue

• Enable EKS Audit Logs
• Query CW Insights (select cluster log group):

fields @timestamp
| filter @logStream like /kube-apiserver-audit/
| filter @message like /seconds after warning threshold/
| parse @message "subject: *, seconds after warning threshold:*\"" as subject, elapsedtime

[ashie]

In my understanding, the fix in fluent-plugin-kubernetes_metadata_filter v2.11.1 is just a work around, not fully resolve the issue in desired way.
It refreshes a token only when it's really expired. See also:
fabric8io/fluent-plugin-kubernetes_metadata_filter#337 (comment)

[ashie]

So I'm considering to help releasing new version of kubeclient (ManageIQ/kubeclient#561 (comment)) but I also don't have enough time for it now...

[oganekoi]

@ashie
Thank you for the clear explanation, I understand the current situation very well.
I am relieved to hear that the tokens are refreshed when they expire and the operation itself seems to be fine.
I understand that you are developing a kubeclient to refresh old tokens. We will continue to monitor this issue closely.
I will close this issue. Thank you very much.

[PettitWesley]

Hey, I implemented the workaround. What was stated in this issue is correct. It refreshes reactively, only when the token is expired.

[ashie]

Thanks following up 👍

[jorge-fabric]

Hello @PettitWesley @ashie 👋 the token still refreshes reactively, correct? So there is no fix to avoid the stale token warnings in EKS for now?

Btw, I'm running an EKS 1.21 and the following fluentd plugins versions:

2022-10-21 18:00:08 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluentd.conf"
2022-10-21 18:00:08 +0000 [info]: gem 'fluentd' version '1.15.2'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.13.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-10-21 18:00:08 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'

[PettitWesley]

@jorge-gyant Yea, the change I contributed (in 2.11.1 IIRC) is reactive.

There was some discussion that making a change in the underlying ruby kube client library would allow proactive refreshes, I am not sure if anyone has worked on that.

[ashie]

kubeclient 4.10.0 supports refreshing the token on every request (when bearer_token_file is set), so that we can fix the issue by updating kubeclient. We'll update it in the next release.

[whereisaaron]

@jorge-gyant I had the same problem, even with latest fluentd, so I switched to fluentbit as the k8s client there can renew tokens before they expire. Looks like it renews every 12-13 minutes. No more stale token alerts 👏

[2022/10/30 08:52:18] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/30 09:05:08] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/30 09:18:08] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/30 09:30:08] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/30 09:45:08] [ info] [filter:kubernetes:kubernetes.0]  token updated

[dhayhak]

Hi @ashie,

so that we can fix the issue by updating kubeclient. We'll update it in the next release.

Do you know which release will have the kubeclient token fix? I'm currently testing with fluentd-daemonset v1.15.3 and I can't confirm which kubeclient version it is using.

Thanks!

[ashie]

v1.15.3 images use kubeclient 4.10.1.
e.g.)

fluentd-kubernetes-daemonset/docker-image/v1.15/debian-s3/Gemfile.lock

Line 94 in 4497779

kubeclient (4.10.1)