Refresh token/api-key periodically #741

aparamon · 2019-01-30T13:38:39Z

Currently, authorization token/api-key is only initialized on loading config:
https://github.com/kubernetes-client/python-base/blob/bd9a8525e9215f7f01c32a321beb9a605cf0402b/config/kube_config.py#L420
https://github.com/kubernetes-client/python-base/blob/bd9a8525e9215f7f01c32a321beb9a605cf0402b/config/kube_config.py#L510
When working with Amazon EKS via aws-iam-authenticator though, token/api-key expires relatively quickly.

It is proposed to introduce a configurable option to specify token/api-key time-to-live. On API call the time should be checked, and if expired the token/api-key should be refreshed by calling https://github.com/kubernetes-client/python-base/blob/bd9a8525e9215f7f01c32a321beb9a605cf0402b/config/kube_config.py#L350 again.
Alternatively, API client could check for 401 Unauthorized return code and refresh token (at most once per API call).

The text was updated successfully, but these errors were encountered:

marsewe · 2019-03-21T16:02:55Z

See also kubernetes-sigs/aws-iam-authenticator#63 and https://stackoverflow.com/questions/48151388/kubernetes-python-client-authentication-issue. Should the user of the client catch the 401 (and re-authenticate and retry) or is that a responsibility of the kubernetes-client?

jmeickle · 2019-06-07T00:40:13Z

Calling config.load_kube_config() again doesn't seem to recreate a token in my case, even though the corresponding kubectl command would have automatically done so. So I think there's some staleness check here that isn't actually occurring.

houqp · 2019-08-05T01:01:44Z

config for an existing client is cached unfortunately :( After calling load_kube_config, you will have to recreate a new api client to pick up the refreshed token. Currently, the expirationTimestamp returned from aws-iam-authenticator is ignored by KubeConfigLoader.

wing328 · 2019-08-08T01:40:58Z

@houqp I wonder if you can send the patch upstream to OpenAPI Generator as that's what the project going to use moving forward to generate API clients: kubernetes-client/gen#93

houqp · 2019-08-08T05:14:11Z

Good call @wing328, do you know if #738 and kubernetes-client/gen#97 are the right PRs to watch?

houqp · 2019-08-08T20:15:02Z

@wing328 i have ported the patches to openapi-generator, could you help review them?

roycaihw · 2019-08-27T17:50:45Z

/assign

roycaihw · 2019-09-18T22:01:42Z

kubernetes-client/gen#97 is merged. We used openapi-generator in 11.0.0a1 release #931

jdamata · 2019-11-13T20:37:43Z

I'm using 11.0.0a1 but still getting intermittent 401's

{"asctime": "2019-11-13 14:38:33,384", "name": "k8s", "levelname": "INFO", "message": "Sleeping for 60 seconds"}
Traceback (most recent call last):
  File "src/main.py", line 57, in <module>
    drain.run()
  File "/opt/git/stratus/stratus-eks/worker_node_rolling_update/src/k8s.py", line 42, in run
    self.drain()
  File "/opt/git/stratus/stratus-eks/worker_node_rolling_update/src/k8s.py", line 33, in drain
    if check_node_drained(self.v1api, node):
  File "/opt/git/stratus/stratus-eks/worker_node_rolling_update/src/k8s_helper.py", line 117, in check_node_drained
    node_body = api.read_node(node_name)
  File "/Users/damatj/.local/share/virtualenvs/worker_node_rolling_update-n2X_lDEC/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 20565, in read_node
    (data) = self.read_node_with_http_info(name, **kwargs)  # noqa: E501
  File "/Users/damatj/.local/share/virtualenvs/worker_node_rolling_update-n2X_lDEC/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 20649, in read_node_with_http_info
    collection_formats=collection_formats)
  File "/Users/damatj/.local/share/virtualenvs/worker_node_rolling_update-n2X_lDEC/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
    _preload_content, _request_timeout)
  File "/Users/damatj/.local/share/virtualenvs/worker_node_rolling_update-n2X_lDEC/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 166, in __call_api
    _request_timeout=_request_timeout)
  File "/Users/damatj/.local/share/virtualenvs/worker_node_rolling_update-n2X_lDEC/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 356, in request
    headers=headers)
  File "/Users/damatj/.local/share/virtualenvs/worker_node_rolling_update-n2X_lDEC/lib/python3.7/site-packages/kubernetes/client/rest.py", line 241, in GET
    query_params=query_params)
  File "/Users/damatj/.local/share/virtualenvs/worker_node_rolling_update-n2X_lDEC/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '96bc2da9-40a0-42b8-9303-3a99b5d77db1', 'Content-Type': 'application/json', 'Date': 'Wed, 13 Nov 2019 19:39:33 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

athom · 2019-12-11T05:39:34Z

11.0.0b2 also not work

   File "/etc/airflow/.local/share/virtualenvs/airflow-bTdwlyD1/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 168, in read_pod
     return self._client.read_namespaced_pod(pod.name, pod.namespace)
   File "/etc/airflow/.local/share/virtualenvs/airflow-bTdwlyD1/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py", line 19078, in read_namespaced_pod
     (data) = self.read_namespaced_pod_with_http_info(name, namespace, **kwargs)  # noqa: E501
   File "/etc/airflow/.local/share/virtualenvs/airflow-bTdwlyD1/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py", line 19169, in read_namespaced_pod_with_http_info
     collection_formats=collection_formats)
   File "/etc/airflow/.local/share/virtualenvs/airflow-bTdwlyD1/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 335, in call_api
     _preload_content, _request_timeout)
   File "/etc/airflow/.local/share/virtualenvs/airflow-bTdwlyD1/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 166, in __call_api
     _request_timeout=_request_timeout)
   File "/etc/airflow/.local/share/virtualenvs/airflow-bTdwlyD1/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 356, in request
     headers=headers)
   File "/etc/airflow/.local/share/virtualenvs/airflow-bTdwlyD1/lib/python3.6/site-packages/kubernetes/client/rest.py", line 241, in GET
     query_params=query_params)
   File "/etc/airflow/.local/share/virtualenvs/airflow-bTdwlyD1/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
     raise ApiException(http_resp=r)
 kubernetes.client.rest.ApiException: (401)
 Reason: Unauthorized
 HTTP response headers: HTTPHeaderDict({'Audit-Id': '685e7f66-8303-418c-9d1d-b82920ef2d0f', 'Content-Type': 'application/json', 'Date': 'Wed, 11 Dec 2019 04:12:24 GMT', 'Content-Length': '129'})
 HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

fejta-bot · 2020-03-10T06:17:31Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

tbarrella · 2020-03-10T06:25:43Z

/remove-lifecycle stale

roycaihw · 2020-03-10T23:39:58Z

OpenAPITools/openapi-generator#3594 was included in openapi-generator 4.1.1. Currently we are still using openapi-generator 3.3.4. We are tracking upgrading openapi-generator in the next major release: #1052, #1088

kubernetes-client/python#741

fejta-bot · 2020-06-08T23:43:49Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

dinvlad · 2020-06-09T03:10:21Z

/remove-lifecycle stale

Uznick · 2021-03-10T11:28:38Z

We have a similiar workaround as well. Works fine for now.

dinvlad · 2021-03-10T18:21:36Z

@houqp by looking at your code, it uses the default load_incluster_config() when running inside the cluster. Does it mean you only experience this issue with the default load_kube_config()? We're seeing it even with load_incluster_config(), if I'm reading that right.

Or maybe our "hanging" issue is actually not due to expired creds, but instead because of the TCP keepalive problem you mentioned there as well (thanks for indirectly bringing this to my attention!) 🤔

dinvlad · 2021-03-10T18:57:46Z

To more specifically expand on our issue (which might be slightly different from others' use of config loader here), we use load_incluster_config() for in-cluster cleanup operation, which in our case is the following pseudo-code:

from kubernetes.watch import Watch

def cleanup(namespace: str):
    """
    Watches for and deletes terminated Jobs.
    This function would not normally be needed if `ttlSecondsAfterFinished` worked.
    However, that feature is currently hidden behind an `alpha` flag in GKE.
    """
    load_incluster_config()
    api = BatchV1Api()
    events = Watch().stream(api.list_namespaced_job, namespace)
    for event in events:
        # cleanup logic

So our issue (if I'm not mistaken) is that no matter how credentials are fetched (either in-cluster or not), they seem to be "cached" in the config, and then stream() generator expires. I wonder if there's a more general way to implement a config client that refreshes credentials automatically.

Please correct me if that's what's already happening in load_incluster_config(), and our issue (watcher stops working after a while) is caused by something else (like the keepalive problem etc.)

houqp · 2021-03-10T19:12:49Z

@dinvlad my fix was only for out of cluster communication where auth goes through iam-authenticator. IIRC, in cluster communication uses builtin K8S auth, which doesn't have this problem.

alexcristi · 2021-04-27T15:16:35Z

Hi there! Any updates on this?

kwlzn · 2021-07-20T02:14:51Z

is anyone actively working on this issue? if not, my team at Twitter might try to take a stab at it soon.

…lient v11.0.0) This is a partial fix for kubernetes-client/python#741, based on the version of this repo included in `kubernetes-client` v11.0.0 (https://github.com/kubernetes-client/python/tree/v11.0.0). As described in kubernetes-client/python#741, some of the authentication schemes supported by Kubernetes require updating the client's credentials from time to time. The Kubernetes Python client currently does not support this, except for when using the `gcp` auth scheme. This is because the OpenAPI-generated code does not generally expect credentials to change after the client is configured. However, in OpenAPITools/openapi-generator#3594, the OpenAPI-generated code added a (undocumented) hook on the `Configuration` object which provides a method for the client credentials to be refreshed as needed. Unfortunately, this version of the Kubernetes client is too old to have that hook, but this patch adds it with a subclass of `Configuration`. Then the `load_kube_config()` function, used by the Kubernetes API to set up the `Configuration` object from the client's local k8s config, just needs to be updated to take advantage of this hook. This patch does this for `exec`-based authentication, which is a partial fix for kubernetes-client/python#741. The plan is to follow up to support this for all other authentication schemes which may require refreshing credentials. The follow-up patch will be based on the latest Kubernetes client and won't need the `Configuration` subclass. As noted above, `load_kube_config()` already has a special-case monkeypatch to refresh GCP tokens. I presume this functionality was added before the OpenAPI generator added support for the refresh hook. A complete fix will probably include refactoring the GCP token refreshing to use the new hook.

This is a fix for kubernetes-client/python#741. As described in kubernetes-client/python#741, some of the authentication schemes supported by Kubernetes require updating the client's credentials from time to time. The Kubernetes Python client currently does not support this, except for when using the `gcp` auth scheme. This is because the OpenAPI-generated client code does not generally expect credentials to change after the client is configured. However, in OpenAPITools/openapi-generator#3594, the OpenAPI generator added a (undocumented) hook on the `Configuration` object which provides a method for the client credentials to be refreshed as needed. Now that this hook exists, the `load_kube_config()` function, used by the Kubernetes API to set up the `Configuration` object from the client's local k8s config, just needs to be updated to take advantage of this hook. This patch does this for `exec`-based authentication, which should resolve kubernetes-client/python#741. Also, as noted above, `load_kube_config()` already has a special-case monkeypatch to refresh GCP tokens. I presume this functionality was added before the OpenAPI generator added support for the refresh hook. This patch also refactors the GCP token refreshing code to use the new hook instead of the monkeypatch. Tests are also updated.

k8s-triage-robot · 2021-10-18T02:25:45Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

dinvlad · 2021-10-18T17:11:50Z

/remove-lifecycle stale

This is a fix for kubernetes-client/python#741. As described in kubernetes-client/python#741, some of the authentication schemes supported by Kubernetes require updating the client's credentials from time to time. The Kubernetes Python client currently does not support this, except for when using the `gcp` auth scheme. This is because the OpenAPI-generated client code does not generally expect credentials to change after the client is configured. However, in OpenAPITools/openapi-generator#3594, the OpenAPI generator added a (undocumented) hook on the `Configuration` object which provides a method for the client credentials to be refreshed as needed. Now that this hook exists, the `load_kube_config()` function, used by the Kubernetes API to set up the `Configuration` object from the client's local k8s config, just needs to be updated to take advantage of this hook. This patch does this for `exec`-based authentication, which should resolve kubernetes-client/python#741. Also, as noted above, `load_kube_config()` already has a special-case monkeypatch to refresh GCP tokens. I presume this functionality was added before the OpenAPI generator added support for the refresh hook. This patch also refactors the GCP token refreshing code to use the new hook instead of the monkeypatch. Tests are also updated.

…32210) This PR adds retries on create/get requests from the test driver to the K8s API when 401 Unauthorized error is encountered. K8S python library expects the ApiClient to be cycled on auth token refreshes. The problem is described in kubernetes-client/python#741. Currently we don't have any hypotheses why we weren't affected by this problem before. To force the ApiClient to pick up the new credentials, I shut down the current client, create a new one, and replace api_client properties on all k8s APIs we manage. This should also work with the Watch-based log collector recovering from an error. To support that, I replace default Configuration so that the next time Watch creates ApiClient implicitly, the Configuration with updated token will be used.

…rpc#32210) This PR adds retries on create/get requests from the test driver to the K8s API when 401 Unauthorized error is encountered. K8S python library expects the ApiClient to be cycled on auth token refreshes. The problem is described in kubernetes-client/python#741. Currently we don't have any hypotheses why we weren't affected by this problem before. To force the ApiClient to pick up the new credentials, I shut down the current client, create a new one, and replace api_client properties on all k8s APIs we manage. This should also work with the Watch-based log collector recovering from an error. To support that, I replace default Configuration so that the next time Watch creates ApiClient implicitly, the Configuration with updated token will be used.

…32210) This PR adds retries on create/get requests from the test driver to the K8s API when 401 Unauthorized error is encountered. K8S python library expects the ApiClient to be cycled on auth token refreshes. The problem is described in kubernetes-client/python#741. Currently we don't have any hypotheses why we weren't affected by this problem before. To force the ApiClient to pick up the new credentials, I shut down the current client, create a new one, and replace api_client properties on all k8s APIs we manage. This should also work with the Watch-based log collector recovering from an error. To support that, I replace default Configuration so that the next time Watch creates ApiClient implicitly, the Configuration with updated token will be used.

This is a fix for kubernetes-client/python#741. As described in kubernetes-client/python#741, some of the authentication schemes supported by Kubernetes require updating the client's credentials from time to time. The Kubernetes Python client currently does not support this, except for when using the `gcp` auth scheme. This is because the OpenAPI-generated client code does not generally expect credentials to change after the client is configured. However, in OpenAPITools/openapi-generator#3594, the OpenAPI generator added a (undocumented) hook on the `Configuration` object which provides a method for the client credentials to be refreshed as needed. Now that this hook exists, the `load_kube_config()` function, used by the Kubernetes API to set up the `Configuration` object from the client's local k8s config, just needs to be updated to take advantage of this hook. This patch does this for `exec`-based authentication, which should resolve kubernetes-client/python#741. Also, as noted above, `load_kube_config()` already has a special-case monkeypatch to refresh GCP tokens. I presume this functionality was added before the OpenAPI generator added support for the refresh hook. This patch also refactors the GCP token refreshing code to use the new hook instead of the monkeypatch. Tests are also updated.

aparamon changed the title ~~Refresh auth token periodically~~ Refresh token/api-key periodically Jan 30, 2019

marsewe mentioned this issue Mar 21, 2019

Retry API call on token expiration #791

Closed

garretthoffman mentioned this issue Mar 26, 2019

Authentication Issue for AWS EKS Cluster kubernetes-client/python-base#125

Closed

houqp mentioned this issue Aug 6, 2019

[Python] support api key refresh in configuration module swagger-api/swagger-codegen#9624

Merged

4 tasks

This was referenced Aug 8, 2019

[Python] support api key refresh in configuration module OpenAPITools/openapi-generator#3594

Merged

[AIRFLOW-5117] support refreshing EKS api tokens apache/airflow#5731

Merged

k8s-ci-robot assigned roycaihw Aug 27, 2019

dgzlopes mentioned this issue Mar 2, 2020

Error Not Handled by K8s Module powerfulseal/powerfulseal#112

Open

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 10, 2020

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 10, 2020

dinvlad added a commit to broadinstitute/dsp-appsec-infrastructure-apps that referenced this issue Apr 6, 2020

[batch] Re-create Batch Api client on every request

5019a67

kubernetes-client/python#741

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 8, 2020

surajkota mentioned this issue May 6, 2021

Create new k8s client everytime to avoid token refresh issues aws-controllers-k8s/test-infra#18

Merged

casassg mentioned this issue Jul 21, 2021

feat(sdk): Surface kubernetes configuration in container builder kubeflow/pipelines#6095

Merged

emenendez mentioned this issue Aug 27, 2021

Refresh exec-based API credentials when they expire kubernetes-client/python-base#250

Merged

oavdeev mentioned this issue Sep 17, 2021

K8S: auth and pod/job race Netflix/metaflow#704

Merged

duferdev mentioned this issue Sep 22, 2021

FEATURE: Refresh aws token LamaAni/KubernetesJobOperator#54

Open

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 18, 2021

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 18, 2021

k8s-ci-robot closed this as completed in kubernetes-client/python-base#250 Nov 30, 2021

seanpang-vmware mentioned this issue Jan 26, 2022

ws_client.py uses select.poll without checking module existence #1686

Closed

kafonek mentioned this issue May 23, 2022

Refresh service account token periodically tomplus/kubernetes_asyncio#207

Closed

AlexisZam mentioned this issue Nov 8, 2022

Requests from outside the cluster may have stale tokens and fail with status code 401 #1946

Closed

sergiitk mentioned this issue Jan 27, 2023

xDS interop: Refresh k8s auth on 401 to pick up updated auth token grpc/grpc#32210

Merged

harryharpel mentioned this issue May 19, 2023

Refresh exec-based API credentials when they expire tomplus/kubernetes_asyncio#257

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refresh token/api-key periodically #741

Refresh token/api-key periodically #741

aparamon commented Jan 30, 2019 •

edited

marsewe commented Mar 21, 2019 •

edited

jmeickle commented Jun 7, 2019

houqp commented Aug 5, 2019

wing328 commented Aug 8, 2019

houqp commented Aug 8, 2019

houqp commented Aug 8, 2019

roycaihw commented Aug 27, 2019

roycaihw commented Sep 18, 2019

jdamata commented Nov 13, 2019

athom commented Dec 11, 2019

fejta-bot commented Mar 10, 2020

tbarrella commented Mar 10, 2020

roycaihw commented Mar 10, 2020

fejta-bot commented Jun 8, 2020

dinvlad commented Jun 9, 2020

Uznick commented Mar 10, 2021

dinvlad commented Mar 10, 2021 •

edited

dinvlad commented Mar 10, 2021

houqp commented Mar 10, 2021

alexcristi commented Apr 27, 2021

kwlzn commented Jul 20, 2021

k8s-triage-robot commented Oct 18, 2021

dinvlad commented Oct 18, 2021

Refresh token/api-key periodically #741

Refresh token/api-key periodically #741

Comments

aparamon commented Jan 30, 2019 • edited

marsewe commented Mar 21, 2019 • edited

jmeickle commented Jun 7, 2019

houqp commented Aug 5, 2019

wing328 commented Aug 8, 2019

houqp commented Aug 8, 2019

houqp commented Aug 8, 2019

roycaihw commented Aug 27, 2019

roycaihw commented Sep 18, 2019

jdamata commented Nov 13, 2019

athom commented Dec 11, 2019

fejta-bot commented Mar 10, 2020

tbarrella commented Mar 10, 2020

roycaihw commented Mar 10, 2020

fejta-bot commented Jun 8, 2020

dinvlad commented Jun 9, 2020

Uznick commented Mar 10, 2021

dinvlad commented Mar 10, 2021 • edited

dinvlad commented Mar 10, 2021

houqp commented Mar 10, 2021

alexcristi commented Apr 27, 2021

kwlzn commented Jul 20, 2021

k8s-triage-robot commented Oct 18, 2021

dinvlad commented Oct 18, 2021

aparamon commented Jan 30, 2019 •

edited

marsewe commented Mar 21, 2019 •

edited

dinvlad commented Mar 10, 2021 •

edited