Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using DynamoDB as a backend, Vault is unhealthy when the standby instance is going down #4876

Closed
odedpriva opened this issue Jul 9, 2018 · 2 comments
Labels
bug Used to indicate a potential bug storage/dynamodb

Comments

@odedpriva
Copy link

Describe the bug
When using DynamoDB as a backend, Vault is unhealthy when the standby instance is going down

To Reproduce
Steps to reproduce the behavior:

  1. Start an HA mode vault using DynamoDB as a backend
  2. Stop the standby instance.
  3. The active vault instance is returning 429 and vault is unavailable till an instance grabs the lock.

This cluster is running on AWS with ELB to run health checks and point traffic to the active instance using v1/sys/health route.

We had downtime with vault when the standby vault ec2 instance had networking issues.

Expected behavior
standby instances health should not affect the healthiness of the cluster.

Environment:

Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0
Unseal Nonce:
Version: 0.8.0
Cluster Name: ****
Cluster ID: 0792fc39-3392-79b5-43b7-c288405a0932

High-Availability Enabled: true
	Mode: active
	Leader Cluster Address: https://***.:8201
  • Vault CLI Version (retrieve with vault version):

vault version
Vault v0.8.0 ('af63d879130d2ee292f09257571d371100a513eb')

  • Server Operating System/Architecture:

                        +-------------+         +------------------+
                        | VAULT 0.8.0 +---------+                  |
                        +------+------+         |                  |
                              |                |                  |
    +-----------+       +------+-------+        |                  |
    |           |       | VAULT 0.8.0  +------- |     DynamoDB     |
    |  AWS ELB  ++      +------+-------+        |                  |
    |           ||             |                |                  |
    +------------+     +-------+--------+       |                  |
                +-+--^+  VAULT 0.8.0  |--------+                  |
                  |   +----------------+       +------------------+
                  |
      +------------+----------------+
      |   Ping Target	              |
      |   HTTP:8200/v1/sys/health   |
      |   Timeout	3 seconds         |
      |   Interval	5 seconds       |
      |   Unhealthy threshold	2     |
      |   Healthy threshold	2       |
      +-----------------------------+
    

Vault server configuration file(s):

storage "dynamodb" {
  ha_enabled = "true"
  region     = "eu-west-1"
  table      = "staging-vault-application"
  redirect_addr = "http://vault.*.com.:8200"
  api_addr = "http://vault.*.com.:8200"
  cluster_addr = "http://vault-internal.*com.:8201"
}


listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 1
}

telemetry {
  statsd_address = "statsd.*.com:8125"
}
@jhmartin
Copy link
Contributor

Possibly related to #5828 ?

@catsby catsby added storage/dynamodb bug Used to indicate a potential bug labels Nov 8, 2019
@michelvocks
Copy link
Contributor

Closing this since #6512 has been merged to fix this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Used to indicate a potential bug storage/dynamodb
Projects
None yet
Development

No branches or pull requests

4 participants