Vault Agent Sidecar With Spring Cloud Vault Not Working

[ChrisJBurns]

Describe the bug
Using version 2.2.7.RELEASE of spring-cloud-starter-vault-config and I have got the following bootstrap.yaml

spring:
  cloud:
    vault:
      kv:
        enabled: true
      authentication: NONE

When I deploy it into Kubernetes, with the following annotations on the Deployment

      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'bot-sa'
        vault.hashicorp.com/namespace: "development"
        vault.hashicorp.com/agent-init-first: "true"

I can see that the vault-init starts, then the vault-agent indeed starts and runs. When the Spring App runs, I see the following

{"@timestamp":"2022-11-18T19:17:57.910Z","@version":"1","message":"[RequestedSecret [path='secrets/dev-team/bot', mode=RENEW]] Lease [leaseId='null', leaseDuration=PT0S, renewable=false] I/O error on GET request for \"https://localhost:8200/v1/secrets/dev-team/bot\": Connect to localhost:8200 [localhost/127.0.0

Ignoring the issue that the KV Engine is Version 2 and for some reason isn't recognising it (have raised this here spring-projects/spring-vault#738), but it doesn't seem to be connecting to the Vault Agent. Tbh, I'm not entirely sure how the whole NONE authentication stuff works, because there isn't any actual code examples of this on any docs - apart from just saying to use NONE. Am I misunderstanding the entire thing, or is the above a genuine issue?

Previous to this, we were using Kubernetes authentication to retrieve the secrets, but we want to cut down the code by just using the Vault Agent - but are having the above problems.

To note: There is nothing wrong with the injection of secrets into the main container, so we know that in terms of injecting secrets using the Vault Secret Injection Template works. We just can't figure out why Spring itself can't pull the secrets through the Vault Agent.

[ChrisJBurns]

Ok, so, it seems to be somewhat working now after spamming as many combinations of the Vault Agent Annotations as possible.

For the record and also other's who are reading this thread, to enable the above to work I had to do 2 things.

The addition of the agent-cache-enable annotation to true. I did try a bunch of others:

vault.hashicorp.com/agent-inject-token: "true"
vault.hashicorp.com/agent-cache-listener-port: "8200"
vault.hashicorp.com/agent-cache-use-auto-auth-token: "force"

But the agent-cache-enable was the only annotation that made any effect.

Secondly, I also noticed that there were SSL problems when connecting to Agent because it was doing a secure request via https://localhost:8200. To stop this, I just added scheme in the bootstrap config, and it ended up looking like:

spring:
  cloud:
    vault:
      kv:
        enabled: true
      scheme: http
      authentication: NONE

Realistically, I should probably set the uri instead to http://localhost:8200 just so it's completely explicit - but am happy that it works for now.

Lastly the annotations used where the following:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: bot-test
  namespace: development
  labels:
    app: bot-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bot-test
  template:
    metadata:
      labels:
        app: bot-test
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'gitbot-sa'
        vault.hashicorp.com/namespace: "development"
        vault.hashicorp.com/agent-init-first: "true"
        vault.hashicorp.com/agent-cache-enable: "true"

I think it's best that we at least add some documentation around this? As I remember trying to do this on other projects and got no luck and have to give up, I had a bit more time this time around to get it working. I'd love to help others reduce time spent finding the above out the hard way if possible?

[ChrisJBurns]

One last comment. The logs indicate that the KV2 /data/ path isn't being used, but it is quite strange, because it's definitely pulling in secrets from a KV2 engine - not entirely sure what's going on there. Logs for fullness

 {"@timestamp":"2022-11-18T19:59:34.714Z","@version":"1","message":"Located property source: [BootstrapPropertySource {name='bootstrapProperties-Context backend: secrets/dev-team/bot'}]","logger_name":"org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration","thread_name":"main","level":"I