Error Not Handled by K8s Module

[mikeln]

Error return from call to k8s api not handled...e.g.:

No handlers could be found for logger "powerfulseal.k8s.k8s_client"
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/Mikel_Nelson/dev/cloud/clusterops-stuff/powerfulseal/powerfulseal/cli/__main__.py", line 601, in <module>
    start()
  File "/Users/Mikel_Nelson/dev/cloud/clusterops-stuff/powerfulseal/powerfulseal/cli/__main__.py", line 597, in start
    main(sys.argv[1:])
  File "/Users/Mikel_Nelson/dev/cloud/clusterops-stuff/powerfulseal/powerfulseal/cli/__main__.py", line 445, in main
    groups_to_restrict_to = k8s_client.get_nodes_groups()
  File "powerfulseal/k8s/k8s_client.py", line 57, in get_nodes_groups
    nodes = self.list_nodes()
  File "powerfulseal/k8s/k8s_client.py", line 80, in list_nodes
    resp = self.client_corev1api.list_node()
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13220, in list_node
    (data) = self.list_node_with_http_info(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13313, in list_node_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 361, in request
    headers=headers)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 240, in GET
    query_params=query_params)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Mon, 18 Mar 2019 16:39:07 GMT', 'Content-Length': '129', 'Content-Type': 'application/json', 'Connection': 'keep-alive', 'Server': 'nginx/1.10.3 (Ubuntu)'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

NOTE: this is a flakey cluster, that sometimes returns this error. Not a consistent error from kubernetes.

Analysis: Logging error...missing handler

[huytran1ibm]

Hello I want to add, I am also getting this error

Traceback (most recent call last):
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/bin/seal", line 10, in <module>
    sys.exit(start())
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/powerfulseal/cli/__main__.py", line 703, in start
    main(sys.argv[1:])
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/powerfulseal/cli/__main__.py", line 656, in main
    metric_collector=metric_collector
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/powerfulseal/policy/policy_runner.py", line 91, in run
    scenario.execute()
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/powerfulseal/policy/scenario.py", line 59, in execute
    initial_set = self.match()
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/powerfulseal/policy/pod_scenario.py", line 50, in match
    for pod in method(params):
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/powerfulseal/policy/pod_scenario.py", line 88, in match_labels
    selector=selector,
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/powerfulseal/k8s/k8s_inventory.py", line 93, in find_pods
    deployment_name=deployment_name,
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/powerfulseal/k8s/k8s_client.py", line 146, in list_pods
    label_selector=selector,
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 12372, in list_namespaced_pod
    (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs)
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 12472, in list_namespaced_pod_with_http_info
    collection_formats=collection_formats)
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 355, in request
    headers=headers)
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in GET
    query_params=query_params)
  File "/Users/huytran/workspace/github.com/bloomberg/powerfulseal/env/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Mon, 02 Mar 2020 05:10:45 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

Run from : local/outside of cluster
Mode : Autonomous
config:
minSecondsBetweenRuns: 300
maxSecondsBetweenRuns: 300

What i've noticed is that after 30 mins I get this error. Therefore, k8 module were successful in executing the podScenarios before.

I've ran twice, the first run errored out after about 30 mins with 2 successful podScenario executions.

The second run errored out about after 40 mins with 3 successful podScenario executions.

Kubernetes cluster used is IKS (Ibm cloud Kubernetes Service)

[huytran1ibm]

Thank you for the reports! I think this issue is related to token expiration in kubernetes-client/python.

that's also my suspicion

[dgzlopes]

Thank you for the reports!

I think this issue is related to token expiration in kubernetes-client/python. Currently, authorization token/api-key is only initialized on loading config, and some authenticators token/api-key expire relatively quickly.

From reading kubernetes-client/python#741, I see that a fix was merged to support the refresh in OpenAPITools/openapi-generator#3594.

Some PR's from projects that implement this refresh technique: apache/airflow#5731 and DataBiosphere/toil#2869.

[huytran1ibm]

kubernetes-client/python#741

[dgzlopes]

@seeker89, can you label this issue?

[seeker89]

Reading through kubernetes-client/python#741 I understand that 11.0.0b2 is still suffering from the issue?

[seeker89]

This seems to be more of a brute force solution to the problem DataBiosphere/toil#2869 but I guess it would move us forward too.

[dgzlopes]

The next major release (kubernetes-client/python#1124) of K8s Python client should fix this issue.

[dgzlopes]

Still waiting for the next major release. We can track it here: kubernetes-client/python#1052