trust chain is not followed when Kubernetes CAs are intermediate CAs #2160
Comments
|
Is a problem with urllib version. Try to use 1.x urllib version. |
|
I think I'm already using that. If I try to |
|
@eloymg I have the same problem. I am using following kubeconfig file: apiVersion: v1
clusters:
- cluster:
certificate-authority-data: Base64-Encrypted Key
server: https://test.....cloud:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
token: Base 64 TokenAnd following three lines: from kubernetes import client, config
config.load_kube_config('./kube_config')
v1 = client.CoreV1Api()
v1.list_pod_for_all_namespaces(watch=False)[ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1000)'))': /api/v1/pods?watch=False
[ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1000)'))': /api/v1/pods?watch=False
[ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1000)'))': /api/v1/pods?watch=Falseurllib3 version: 1.26.18 |
|
@eloymg I have the same problem too. |
|
After a bit of digging, I found out what the cause of my issue is. The problem occurs when I manually generate my Kubernetes CA certificates as intermediate certificates using a custom CA. I followed this guide to do so. I would like to point out that the Root CA certificate that was used to generate the intermediate CA certificates (as shown in the link above) is trusted by the machine and was placed under |
@brainplot Nice finding! I wonder if you would like to propose a fix? |
|
Hi, After reading rest.py code: # cert_reqs
if configuration.verify_ssl:
cert_reqs = ssl.CERT_REQUIRED
else:
cert_reqs = ssl.CERT_NONEIn your code try : from kubernetes import client, config
config.load_kube_config('./kube_config')
config.verify_ssl=False ## <<< Perhaps can be setup in config
v1 = client.CoreV1Api()
v1.list_pod_for_all_namespaces(watch=False)It works for me (no more SSL issue), my code: configuration = kubernetes.client.Configuration()
# Configure API key authorization: BearerToken
configuration.api_key['authorization'] = 'YOUR_API_KEY'
# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
# configuration.api_key_prefix['authorization'] = 'Bearer'
requests.packages.urllib3.disable_warnings()
# Defining host is optional and default to http://localhost
configuration.host = "https://10.96.0.1"
configuration.verify_ssl=False
# Defining host is optional and default to http://localhost
# Enter a context with an instance of the API kubernetes.client
api_client=kubernetes.client.ApiClient(configuration)
# Create an instance of the API class
api_instance = kubernetes.client.WellKnownApi(api_client) |
|
I understand how that can work but there's no reason why I should disable SSL/TLS verification since my setup has a perfectly valid certificate trust chain. |
|
Same problem here connecting to EKS v1.26 cluster using in-cluster configuration. Tried: Still doesn't work: I'm using Alpine linux v3.19: |
|
Spent a few hours debugging this issue. It appears that the API and client are functioning as expected, but the error message is confusing for users. The issue is caused the size of the configMap. Kubernetes configMaps have a size limit of 1MB. This limit is set by In my case the file was ~12MB, so obviously doesn't fit in a configMap. Here is a sample code to test that configMap creation works: # Import necessary libraries
from kubernetes import client, config
# Load in-cluster configuration
config.load_incluster_config()
# Create a Kubernetes API client
v1 = client.CoreV1Api()
# Define the configmap data
data = {"data": "123"}
# Create the configmap object
configmap = client.V1ConfigMap(
api_version="v1",
kind="ConfigMap",
metadata=client.V1ObjectMeta(
name="sample"
),
data=data
)
# Create the configmap in the cluster
v1.create_namespaced_config_map(namespace="sfuga", body=configmap)
# Print success message
print("Configmap created successfully.") |
|
@atmosx I'm honestly unsure that is relevant here. I had this issue just trying to list pods in my cluster. It's clearly something to do with the certificate the API server serves. |
|
I had the same issue. I solved it by adding the certificate-authority key to my kubeconfig as mentioned in this post : https://stackoverflow.com/questions/48351308/how-to-specify-ca-bundle-in-kubernetes-python-client |
|
@louisgls I no longer need this library thus I don't have a reason to try this. However, thank you for providing a solution. |
|
I'm seeing the same issue when I create a cluster using a single CA certificate as intermediate as described in brainplot's comment . As this is a valid configuration described in Kubernetes' own docs and causes the minimal example described in this project's @brainplot Thanks for your excellent troubleshooting. Would you mind retitling this issue as "client doesn't follow trust chain when using single CA certificate as intermediate" or something of the sort? |
brainplot
changed the title
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
|
Thank you @inflatador. I've updated the title and I believe the new one better describes the issue. If not, we can discuss how to clarify further. |
What happened (please include outputs or screenshots):
I was trying the client to obtain info about the running pods in a freshly-installed Kubernetes cluster using exactly the example provided in the
README.mdbut I was hit with this SSL error:What you expected to happen:
I was expecting the example to work 😄
How to reproduce it (as minimally and precisely as possible):
To be honest, I'm not sure. This is a freshly installed Ubuntu machine with a freshly-installed Kubernetes cluster.
Anything else we need to know?:
The cluster is generating its certificates using a custom CA that all nodes trust (thanks to the
update-ca-certificatesscript), including the one I'm running this on.It should be noted that
kubectlworks perfectly fine with no issues whatsoever!Environment:
Kubernetes version (
kubectl version):OS (e.g., MacOS 10.13.6):
Python version (
python --version)Python client version (
pip list | grep kubernetes)The text was updated successfully, but these errors were encountered: