Option to transparently Write on read if no secret exists

[psych0d0g]

i would love to have a feature to basically mimic the ansible.builtin.password lookup plugin with hashi_vault's lookup plugin:

Exerpt from the password lookup plugin docs:

Generates a random plaintext password and stores it in a file at a given filepath.
If the file exists previously, it will retrieve its contents, behaving just like with_file.

So i would love to be able to just define a lookup somewhere in my role, and if there is no corresponding secret found in hashicorp vault, it should create one automatically for future reference and store it in the supplied "path" in vault, if there is a secret found, just return what is already existing.

that way one could write roles and playbooks without having to ever deal with secrets manually at all, just let the lookup plugin do the generation and storage of secrets in hashicorp vault on demand.

What do you think?

[briantist]

Hi @psych0d0g welcome!

This is an interesting proposal, but there are a few possible issues with it.

First is that generally in lookups, we don't really want to be changing state, if we can help it. The password lookup is an older lookup and I'm not sure that pattern would be accepted today, but also it changes what is meant to be a file dedicated for the purpose, so it's relatively benign.

That being said, we do already have a lookup (vault_write) that rather explicitly performs writes. That lookup was controversial, and I opted to include it because of certain operations in Vault that require a write in order to retrieve information (like generating a new secret ID for an approle).

But even if were to implement such a lookup, there is not a clean way I can think of to select a secret backend, so it would probably be dedicated to a single particular backend like kv2 or kv1 or cubbyhole, making it somewhat less flexible.

In addition, the secret "not existing" could be taken to mean that either the secret path itself doesn't exist, or that the secret exists but a particular key in the secret does not exist. These require different handling and operations, but I'll assume the former for now.

This type of lookup might be something I'd consider accepting a contribution for, but I'd have to give it more thought.

---

In the meantime, there re some alternatives.

You could create your own lookup independently of this collection. It's possible to use the module_utils of a collection from outside the collection, so you could take advantage of the shared code that exists here: https://github.com/ansible-collections/community.hashi_vault/tree/main/plugins/module_utils
But please understand that we explicitly mark these utils as being for internal use, and we do not guarantee API stability for them, nor announce those changes.

You can also create a lookup that internally calls other lookups; for example your lookup plugin could make calls using the existing community.hashi_vault.vault_kv2_get and community.hashi_vault.vault_write lookups. In the case where it needs to generate a password it can even call ansible.builtin.password with a file of /dev/null to do the generation.

In the case of using anything but token or none auth, you should also make a call to community.hashi_vault.vault_login so that the chosen method does cause two separate login events for retrieval and writes, which would generate extra tokens.

This option ends up being less code for you to write, and it would be using stable APIs, but it may not be as straightforward either.

It should also be possible to do this with existing lookups in Jinja2, although it may be a bit cumbersome. Here's an example that doesn't take into account single login, and is not tested at all.

It assumes you have a secret path some_secret with a sub-path that matches the short name of the current host (it could just as easily be shared among all hosts, or interpolated differently).

- hosts: myhosts
  vars:
    ansible_hashi_vault_auth_method: token
    ansible_hashi_vault_token: "{{ my_token }}"
    ansible_hashi_vault_engine_mount_point: secret
    path: "some_secret/{{ inventory_hostname_short }}"
    write_path: "{{ ansible_hashi_vault_engine_mount_point }}/data/{{ path }}"
    secret_key: passwd
    secret: >-
      {{
        lookup('community.hashi_vault.vault_kv2_get', path, errors='ignore')
        | ternary(
          lookup('community.hashi_vault.vault_kv2_get', path).secret[secret_key],
          lookup('community.hashi_vault.vault_write', write_path, data=lookup(
            'ansible.builtin.password', '/dev/null', seed=inventory_hostname_short
          )) | default({}, True) | dict2items | reject('==', 0) | reject('!=', 0) | default(
          lookup('community.hashi_vault.vault_kv2_get', path).secret[secret_key],
          True
          )
        )
      }}
  tasks:
    - ansible.builtin.debug:
        var: secret

There are almost certainly mistakes here, and it makes lots of unnecessary calls; there might be ways to improve this in pure jinja as well. It's obviously really messy, but as you can see once the initial thing is defined, you need only reference it by var to execute it each time since templating is lazy.

[psych0d0g]

If anyone actually wants this feature, ive written an independent ansible custom lookup_plugin now that can transparently create a secret in hashicorp vault if there is none already, and returns the value of an pre-existing secret otherwise.

#!/usr/bin/env python
# hashicorp_vault.py - Ansible plugin for looking up secrets from HashiCorp Vault
#
# description: This Ansible lookup plugin allows users to retrieve secrets from a HashiCorp Vault instance.
# author: Lukas Wingerberg <h@xx0r.eu>
# requirements:
#   - hvac python module
#   - VAULT_TOKEN environment variable set to a valid Vault token
# version: 1.0.0
# license: Apache 2.0
# platforms:
#   - any
# notes:
#   - This plugin requires Python 3.6 or higher.
#   - This plugin uses the "kv2" secrets engine and assumes that the secrets are stored under a "ansible/" mount point.
#     If you use a different mount point or engine, you will need to modify the code accordingly.
#   - This plugin generates random secrets with a length of 10 characters by default. You can override the default
#     length by passing a "length" parameter to the lookup function, like this: "{{ lookup('hashicorp_vault', 'secret/path:key', length=20) }}"

import os
import hvac
import string
import random
from ansible.errors import AnsibleError
from ansible.plugins.lookup import LookupBase

VAULT_API="https://your_vault_host:8200"

class LookupModule(LookupBase):
    """
    An Ansible lookup plugin that retrieves secrets from HashiCorp Vault.

    This plugin requires the `hvac` Python module to be installed.

    The plugin uses the `VAULT_TOKEN` environment variable to authenticate to Vault.
    """

    def run(self, terms, variables=None, **kwargs):
        """
        Retrieve secrets from HashiCorp Vault.

        Args:
            terms (list): A list of terms in the format `path` or `path:key`.
                The `path` parameter specifies the path to the Vault secret, and the `key` parameter (optional)
                specifies the key of the value to retrieve. If the `key` parameter is not provided, the entire
                secret data is returned.

        Returns:
            A list of retrieved secret values.

        Raises:
            AnsibleError: If the `hvac` module is not installed or if the `VAULT_TOKEN` environment variable
                is not set.
        """
        # Check if the hvac module is installed
        if not hvac:
            raise AnsibleError("Vault lookup plugin requires the hvac python module")

        # Get the Vault token from the VAULT_TOKEN environment variable
        token = os.environ.get('VAULT_TOKEN', None)
        if not token:
            raise AnsibleError("VAULT_TOKEN environment variable not set")

        # Create a Vault client and authenticate with the token
        client = hvac.Client(
            url=VAULT_API,
            token=token
        )

        results = []

        # store optional parameter "length" or set to default of 10
        length = kwargs.get('length', 10)

        # Iterate through the terms and retrieve the corresponding secrets
        for term in terms:
            # Parse the term to extract the secret path and key
            secret_path, secret_key = self._parse_term(term)
            # Get the secret value from Vault, or create a new one if it doesn't exist
            secret_value = self._get_secret(client, secret_path, kwargs.get('length', 10), secret_key)
            results.append(secret_value)

        return results

    def _parse_term(self, term):
        """
        Parse a term in the format `path` or `path:key`.

        Args:
            term (str): The term to parse.

        Returns:
            A tuple containing the secret path and key.
        """
        parts = term.split(":")
        if len(parts) == 2:
            return parts[0], parts[1]
        else:
            return term, None

    def _get_secret(self, client, path, length, key=None):
        """
        Retrieve a secret from Vault or create a new one with a random value.

        Args:
            client (hvac.Client): A client object for Vault API communication.
            path (str): The path to the secret.
            length (int): The length of the secret value to be generated if the secret
                doesn't exist.
            key (str, optional): The key of the secret value to retrieve, if the secret
                has multiple keys.

        Returns:
            str: The value of the retrieved or newly created secret.

        Raises:
            AnsibleError: If the secret path does not exist or permission is denied,
                or if any other error occurs during Vault API communication.
        """
        try:
            # Try to retrieve the secret from Vault
            secret = client.secrets.kv.v2.read_secret_version(path=path, mount_point='ansible')
            data = secret.get('data', {}).get('data', None)
            if key:
                return data.get(key, None)
            else:
                return data
        except hvac.exceptions.InvalidPath:
            # If the secret does not exist, create a new one with a random value
            return self._create_secret(client, path, length, key)
        except hvac.exceptions.Forbidden:
            # Permission denied error
            raise AnsibleError(f"Permission denied to read secret at path: {path}")
        except Exception as e:
            # Other errors
            raise AnsibleError(f"Failed to retrieve secret at path {path}: {e}")

    def _create_secret(self, client, path, length, key=None):
        """
        Creates a new secret at the specified path with a randomly generated
        value of the specified length. If the key argument is provided, the
        secret value will be stored under that key. Otherwise, the secret value
        will be stored under the 'value' key.

        Args:
            client (hvac.Client): An authenticated instance of the hvac.Client class
            path (str): The path to the secret in the vault
            length (int): The length of the randomly generated secret value
            key (str, optional): The key to use when storing the secret value. Defaults to None.

        Returns:
            str: The randomly generated secret value that was stored in the vault
        """
        secret_value = self._generate_password(length)
        data = {
            key or 'value': secret_value
        }
        secret = client.secrets.kv.v2.create_or_update_secret(
            path=path,
            secret=data,
            mount_point='ansible'
        )
        return secret_value

    def _generate_password(self, length):
        """
        Generate a random password of the given length.

        Args:
            length (int): The length of the password to generate.

        Returns:
            str: A string containing the randomly generated password.
        """
        chars = string.ascii_letters + string.digits
        return ''.join(random.choice(chars) for i in range(length))