CA verification does not work at all with hvac 2.0 on FreeBSD

[Vladimir-csp]

SUMMARY

After upgrading py39-hvac from 1.2.1 to 2.0.0 vault modules fail with requests.exceptions.SSLError due to CERTIFICATE_VERIFY_FAILED

ISSUE TYPE

• Bug Report

COMPONENT NAME

hvac

ANSIBLE VERSION

ansible [core 2.15.6]
  python version = 3.9.18 (main, Oct 19 2023, 22:57:51) [Clang 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc1238 (/usr/local/bin/python3.9)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

7.5.0

CONFIGURATION

OS / ENVIRONMENT

FreeBSD 13

STEPS TO REPRODUCE

Set variables in context of vault module or lookup:

ansible_hashi_vault_auth_method: token
ansible_hashi_vault_token: ...
ansible_hashi_vault_url: https://...
ansible_hashi_vault_ca_cert: /etc/ssl/certs # FreeBSD trust store, a dir of hashed certs, including a custom CA required for vault URL.

Try accessing vault via any vault module.

EXPECTED RESULTS

It works. Preferably it should work even without ansible_hashi_vault_ca_cert, since /etc/ssl/certs is a system trust store.

ACTUAL RESULTS

Underlying python requests module fails to verify HTTPS server, despite CA exists in system's trust store (hvac 1.2.1), and despite ansible_hashi_vault_ca_cert points it to system trust store explicitly (hvac 2.0.0).

fatal: [localhost]: FAILED! => 
  msg: 'An unhandled exception occurred while running the lookup plugin ''community.hashi_vault.vault_kv2_get''. Error was a <class ''requests.exceptions.SSLError''>, original message: HTTPSConnectionPool(host=''_redacted_'', port=_redacted_): Max retries exceeded with url: /v1/_redacted_ (Caused by SSLError(SSLCertVerificationError(1, ''[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1134)''))). HTTPSConnectionPool(host=''_redacted_'', port=_redacted_): Max retries exceeded with url: /v1/_redacted_ (Caused by SSLError(SSLCertVerificationError(1, ''[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1134)'')))'

[briantist]

Thanks for reporting this @Vladimir-csp , I have been looking at it, but I've been very short on time. I believe the changes, ultimately, are related to the change in hvac: hvac/hvac#1021

And since this collection does use a custom session, we probably have to adjust for those changes. I still don't have a 100% complete picture yet, and I can't actually say if this truly only affects FreeBSD (it shouldn't).

[briantist]

@Vladimir-csp can I ask if you are also using the retries option?

[Vladimir-csp]

Yes, retries are set via ansible_hashi_vault_retries

[briantist]

Thank you @Vladimir-csp , can you also show a dump of your env vars (non-sensitive values of course).

As I look through the code, the changes we made in hvac should not be affecting our use case, even when retries are being used (I asked about that because it's the only time we use a custom session). But of course, the change is suspicious, it lines up, I just can't figure out (yet) how it would end up in your situation.

If you try without retries, does that fix the CA issue or is it still present?

[Vladimir-csp]

I will be able to check in a week+ or so, thanks.

[Vladimir-csp]

Yes, with ansible_hashi_vault_retries: ~ it works with hvac 2.0

(but still requires explicitly setting ansible_hashi_vault_ca_cert: /etc/ssl/certs)

[briantist]

thank you @Vladimir-csp

can you also show a dump of your env vars (non-sensitive values of course)

[Vladimir-csp]

_=/usr/bin/env
ANSIBLE_LOG_PATH=/dev/null
BLOCKSIZE=K
EDITOR=mcedit
ENV=/data/home/username/.shrc
HOME=/data/home/username
LANG=en_GB.UTF-8
LC_ALL=en_GB.UTF-8
LOGNAME=username
MAIL=/var/mail/username
OLDPWD=/data/home/username
PAGER=less
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/data/home/username/bin
PS1=\[\033[01;32m\]\u\[\033[00m\]@\[\033[01;35m\]\h\[\033[00m\]:\[\033[01;34m\]\w\n\[\033[01;00m\]\$ 
PWD=/data/home/username/git/[...]/ansible
SHELL=/usr/local/bin/bash
SHLVL=1
SSH_CLIENT=[...]
SSH_CONNECTION=[...]
SSH_TTY=/dev/pts/0
TERM=xterm-256color
USER=username

there was also VAULT_ADDR=https://..., but it does not make any difference.

[Vladimir-csp]

Are there any news or estimates on this?

[briantist]

Unfortunately not, I started looking at it back then, but I have not been able to find the time since then due to other priorities in the collection and mostly lack of time in general.

I still think this is worthwhile to work on but I can't offer any estimate. If you or anyone would like to try to investigate further or put up a fix I would certainly encourage that too.

[daniel-chamberland]

I wanted to add my experience to this issue. I'm seeing the same behavior when upgrading from hvac 1.2.1 to hvac 2.1.0 or hvac 2.2.0 in an Ubuntu 20.04/22.04 WSL image. We have a Vault setup secured using an internal certificate authority. As part of our Ansible Vault lookups, we passed the internal CA cert using the cacert/ca_cert parameter. This is no longer being respected, and calls to Vault are failing with this message:

(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: 
unable to get local issuer certificate (_ssl.c:1006)'))). 
HTTPSConnectionPool(host='vaultserver.domain.com', port=443): 
Max retries exceeded with url: /v1/engine/data/path/to/credential 
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] 
certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')

The stack trace shows failure with requests, and requests uses certifi package. I'm able to get things working again by adding our internal CA to the certifi cacert.pem located in the certifi package: .local/lib/python3.11/site-packages/certifi/cacert.pem

Certifi doesn't support addition/removal of certificates and this isn't a valid long-term workaround (certifi updates will break this again).

So for now, the only workaround is to keep hvac at version 1.2.1 until this is addressed in community.hashi_vault.

As @briantist pointed out, this issue stems from hvac changes to session

And here's the hvac outline of breaking changes planned for 2.x and 3.x

hvac guidance for making use of a private CA

I think the changes to community.hashi_vault need to be made here, but I'm not familiar enough with python to say for sure (or offer a fix). :(

[briantist]

@daniel-chamberland thanks very much for the additional information, I'm still very short on time until at least later this month, but I do hope to track down the cause and fix, whether it's in this collection or in hvac or both.