fix: change disable_verify_ssl behaviour

[wselfjes]

The problem is that verify_ssl is overwritten by the
configuration from the kube_config or load_incluster_config file.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[dstandish]

hi @wselfjes can you please explain in more detail what the problem is and how your fix resolves it? may need a test too.

[wselfjes]

hi @dstandish
The problem that we found in our company is that when we want to use KubernetesExecutor, airflow crashes with an error:

MaxRetryError: HTTPSConnectionPool(host=‘host’, port=6443): Max retries exceeded with url: /api/v1/pods?watch=False (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1091)')))

We tried to play around with certificates but the problem stayed. Other services work fine.
So we decide to specify verify_ssl = False and AIRFLOW__KUBERNETES__VERIFY_SSL: false in the airflow config.
But it didn’t work the same way as in the documentation.

After some research I found that airflow 2.3.2 has following behaviour with get_kube_client() and _disable_verify_ssl():

1. sets verify_ssl=False in the in-memory kubernetes config
2. reads incluster config that rewrites previous set field.

The result is that airflow config field does not apply.

I propose a solution to change the order of get_kube_client() actions:

1. read incluster config or local ./kube/config file
2. if verify_ssl = False set , set it in in-memory config

[wselfjes]

I don't understand why 2 tests are failing with aborting(maybe timeout): Tests / Python unit tests for helm chart (pull_request) & Tests / Sqlite Py3.7: API Always CLI Core Integration Other Providers WWW (pull_request).
Idk how to fix it on my own.

[potiuk]

I've re-run them - likely an intermittent failure.

[wselfjes]

I've re-run them - likely an intermittent failure.

The test failed again(

[potiuk]

I re-run it again, but maybe it's a sign that the change cause the tests to fail (they apparently aborted) and they need fixing.

[wselfjes]

I re-run it again, but maybe it's a sign that the change cause the tests to fail (they apparently aborted) and they need fixing.

Finally

[boring-cyborg]

Awesome work, congrats on your first merged pull request!