Unexpected environment variables half the time while using secrets-init

[evandam]

We're using secrets-init in combination with godotenv to source environment variables and then inject secrets into the environment.

We're seeing unexpected behavior where secrets-init does not set environment variables properly roughly half the time.

❯ cat .env
ENCRYPTION_KEY=
AWS_REGION=us-west-2
AWS_SECRETS=arn:aws:secretsmanager:us-west-2:123456789012:secret:my-secret

❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

# 12:36:32
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=

# 12:36:34
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

# 12:36:35
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

# 12:36:37
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=

# 12:36:39
❯ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY
ENCRYPTION_KEY=pulled_from_secrets_manager

[evandam]

We also tried to find a pattern by trying to trigger rate limiting from Secrets Manager to see if it was related, but it doesn't look like it. It's very random.

❯ seq 1 50 | xargs -I% -P50 sh -c '{ godotenv -f ".env" -- secrets-init -- env | grep ENCRYPTION_KEY; }'
ERRO[0002] failed to resolve secrets                     error="failed to get secret from AWS Secrets Manager: ProcessProviderExecutionError: error in credential_process\ncaused by: wait: no child processes"
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=
ENCRYPTION_KEY=
ENCRYPTION_KEY=pulled_from_secrets_manager
ENCRYPTION_KEY=pulled_from_secrets_manager

Edit: It seems that the issue is happening if the environment variable has a default value (ex: ENCRYPTION_KEY=), this will sporadically fail.