Vault agent and exit_after_retry_failure + error_on_missing_key

[chr-b]

Describe the bug

Vault agent should exit when accessing a non-existing secret from Vault that does not exist, if the agent configuration specifies num_retries, exit_after_retry_failure, error_on_missing_key and exit-after-auth.

Instead, Vault agent will stop the runner after failing to fetch the non-existing secret, but then creates a new runner that will again fail to fetch the non-existing secret. This will be an infinite loop.

This problem occurs with Agent version 1.8.1. This is working fine (=as expected) with version 1.7.3.

To Reproduce

Vault agent config file:

    pid_file = "/home/vault/pidfile"
    exit_after_auth = true
    vault {
      address = "${VAULT_ADDR}"
      retry {
        num_retries = 3
      }
    }
    auto_auth {
      method "kubernetes" {
        mount_path = "auth/kubernetes"
        config = {
          role = "${ROLENAME}"
        }
      }
      sink "file" {
        config = {
          path = "/home/vault/.vault-token"
        }
      }
    }
    retry {
      num_retries = 3
    }
    template_config {
      exit_after_retry_failure = true
    }
    template {
      error_on_missing_key = true
      destination = "/data/key.pem"
      contents = {{- with secret "secret/data/nodes/1" }}{{ base64Decode .Data.data.key }}{{ end }}
    }

Expected behavior

As the specified secret does not exist, Vault agent should exit with failure after 3 retries when started with vault agent -config /home/vault/vault-agent-config.hcl -exit-after-auth

Environment:

Vault Agent v1.8.1
Version Sha: 4b0264f

Vault agent log file:

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.8.1
             Version Sha: 4b0264f28defc05454c31277cfa6ff63695a458d

2021-08-18T09:12:53.975Z [INFO]  sink.file: creating file sink
2021-08-18T09:12:53.975Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2021-08-18T09:12:53.975Z [INFO]  template.server: starting template server
2021-08-18T09:12:53.975Z [INFO] (runner) creating new runner (dry: false, once: false)
2021-08-18T09:12:53.975Z [INFO] (runner) creating watcher
2021-08-18T09:12:53.975Z [INFO]  auth.handler: starting auth handler
2021-08-18T09:12:53.975Z [INFO]  auth.handler: authenticating
2021-08-18T09:12:53.976Z [INFO]  sink.server: starting sink server
2021-08-18T09:12:54.015Z [INFO]  auth.handler: authentication successful, sending token to sinks
2021-08-18T09:12:54.015Z [INFO]  auth.handler: starting renewal process
2021-08-18T09:12:54.016Z [INFO]  sink.file: token written: path=/home/vault/.vault-token
2021-08-18T09:12:54.016Z [INFO]  sink.server: sink server stopped
2021-08-18T09:12:54.016Z [INFO]  sinks finished, exiting
2021-08-18T09:12:54.016Z [INFO]  template.server: template server received new token
2021-08-18T09:12:54.016Z [INFO] (runner) stopping
2021-08-18T09:12:54.016Z [INFO] (runner) creating new runner (dry: false, once: false)
2021-08-18T09:12:54.016Z [INFO] (runner) creating watcher
2021-08-18T09:12:54.016Z [INFO] (runner) starting
2021-08-18T09:12:54.027Z [INFO]  auth.handler: renewed auth token
2021-08-18T09:12:54.029Z [WARN] (view) vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault (retry attempt 1 after "250ms")
2021-08-18T09:12:54.283Z [WARN] (view) vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault /data/fabric-nodes/vault (retry attempt 2 after "500ms")
2021-08-18T09:12:54.787Z [WARN] (view) vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault /data/fabric-nodes/vault (retry attempt 3 after "1s")
2021-08-18T09:12:55.790Z [ERR] (view) vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault /data/fabric-nodes/vault (exceeded maximum retries)
2021-08-18T09:12:55.790Z [ERR] (runner) watcher reported error: vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault
2021-08-18T09:12:55.790Z [ERROR] template.server: template server error: error="vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault"
2021-08-18T09:12:55.790Z [INFO] (runner) stopping
2021-08-18T09:12:55.790Z [INFO] (runner) creating new runner (dry: false, once: false)
2021-08-18T09:12:55.791Z [INFO] (runner) creating watcher
2021-08-18T09:12:55.791Z [INFO] (runner) starting
2021-08-18T09:12:55.800Z [WARN] (view) vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault (retry attempt 1 after "250ms")
2021-08-18T09:12:56.052Z [WARN] (view) vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault  (retry attempt 2 after "500ms")
2021-08-18T09:12:56.554Z [WARN] (view) vault.read(secret/data/nodes/vault): nvault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault  (retry attempt 3 after "1s")
2021-08-18T09:12:57.556Z [ERR] (view) vault.read(secret/data/nodes/vault): vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault  (exceeded maximum retries)
2021-08-18T09:12:57.556Z [ERR] (runner) watcher reported error: vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault
2021-08-18T09:12:57.556Z [ERROR] template.server: template server error: error="vault.read(secret/data/nodes/vault): no secret exists at secret/data/nodes/vault"
2021-08-18T09:12:57.556Z [INFO] (runner) stopping
2021-08-18T09:12:57.556Z [INFO] (runner) creating new runner (dry: false, once: false)
2021-08-18T09:12:57.556Z [INFO] (runner) creating watcher
2021-08-18T09:12:57.556Z [INFO] (runner) starting
...

Additional context

I wonder if this is possibly related to the following changes:

• agent: allow templating to indefinitely retry if explicitly set to zero #11711
• vault agent - respect consul-template retry configuration #9059

[calvn]

I see that we mention exit_after_retry_failure on the docs, but it should be exit_on_retry_failure. Can you give that a try (I also don't see that being provided in the config)? Note that you need to specify both in order for an exit to occur.

[chr-b]

Hi @calvn ,

I can confirm that exit_on_retry_failure fixes the problem.