Skip to content

Commit

Permalink
Improve "Integrated Storage" documentation (hashicorp#12200)
Browse files Browse the repository at this point in the history
* Improve "Integrated Storage" documentation

* add missing markup
* add more links to the configuration pages

* Improve the Raft Storage configuration page

* More markup

* Improve the "High Availability" documentation

* More links to the configuration pages

* More links

* even more links
  • Loading branch information
multani authored and jartek committed Sep 11, 2021
1 parent d7e22b1 commit 625dbd8
Show file tree
Hide file tree
Showing 3 changed files with 177 additions and 130 deletions.
116 changes: 70 additions & 46 deletions website/content/docs/concepts/ha.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,21 +21,24 @@ information is also available on the
To be highly available, one of the Vault server nodes grabs a lock within the
data store. The successful server node then becomes the active node; all other
nodes become standby nodes. At this point, if the standby nodes receive a
request, they will either forward the request or redirect the client depending
on the current configuration and state of the cluster -- see the sections below
for details. Due to this architecture, HA does not enable increased
scalability. In general, the bottleneck of Vault is the data store itself, not
Vault core. For example: to increase the scalability of Vault with Consul, you
would generally scale Consul instead of Vault.
request, they will either [forward the request](#request-forwarding) or
[redirect the client](#client-redirection) depending on the current
configuration and state of the cluster -- see the sections below for details.
Due to this architecture, HA does not enable increased scalability. In general,
the bottleneck of Vault is the data store itself, not Vault core. For example:
to increase the scalability of Vault with Consul, you would generally scale
Consul instead of Vault.

Certain storage backends can support high availability mode, which enable them
to store both Vault's information in addition to the HA lock. However, Vault
also supports split data/HA mode, whereby the lock value and the rest of the
data live separately. This can be done by specifying both the `storage` and
`ha_storage` stanzas in the configuration file with different backends. For
instance, a Vault cluster can be set up to use Consul as the `ha_storage` to
manage the lock, and use Amazon S3 as the `storage` for all other persisted
data.
data live separately. This can be done by specifying both the
[`storage`](/docs/configuration#storage) and
[`ha_storage`](/docs/configuration#ha_storage) stanzas in the configuration file
with different backends. For instance, a Vault cluster can be set up to use
Consul as the [`ha_storage`](/docs/configuration#ha_storage) to manage the lock,
and use Amazon S3 as the [`storage`](/docs/configuration#storage) for all other
persisted data.

The sections below explain the server communication patterns and each type of
request handling in more detail. At a minimum, the requirements for redirection
Expand Down Expand Up @@ -84,28 +87,36 @@ always required for all HA setups.

Some HA data store drivers can autodetect the redirect address, but it is often
necessary to configure it manually via a top-level value in the configuration
file. The key for this value is `api_addr` and the value can also be specified
by the `VAULT_API_ADDR` environment variable, which takes precedence.
file. The key for this value is [`api_addr`](/docs/configuration#api_addr) and
the value can also be specified by the `VAULT_API_ADDR` environment variable,
which takes precedence.

What the `api_addr` value should be set to depends on how Vault is set up.
There are two common scenarios: Vault servers accessed directly by clients, and
Vault servers accessed via a load balancer.
What the [`api_addr`](/docs/configuration#api_addr) value should be set to
depends on how Vault is set up. There are two common scenarios: Vault servers
accessed directly by clients, and Vault servers accessed via a load balancer.

In both cases, the `api_addr` should be a full URL including scheme
(`http`/`https`), not simply an IP address and port.
In both cases, the [`api_addr`](/docs/configuration#api_addr) should be a full
URL including scheme (`http`/`https`), not simply an IP address and port.

### Direct Access

When clients are able to access Vault directly, the `api_addr` for each
node should be that node's address. For instance, if there are two Vault nodes
`A` (accessed via `https://a.vault.mycompany.com:8200`) and `B` (accessed via
`https://b.vault.mycompany.com:8200`), node `A` would set its `api_addr`
to `https://a.vault.mycompany.com:8200` and node `B` would set its
`api_addr` to `https://b.vault.mycompany.com:8200`.
When clients are able to access Vault directly, the
[`api_addr`](/docs/configuration#api_addr) for each node should be that node's
address. For instance, if there are two Vault nodes:

* `A`, accessed via `https://a.vault.mycompany.com:8200`
* `B`, accessed via `https://b.vault.mycompany.com:8200`

Then node `A` would set its
[`api_addr`](/docs/configuration#api_addr) to
`https://a.vault.mycompany.com:8200` and node `B` would set its
[`api_addr`](/docs/configuration#api_addr) to
`https://b.vault.mycompany.com:8200`.

This way, when `A` is the active node, any requests received by node `B` will
cause it to redirect the client to node `A`'s `api_addr` at
`https://a.vault.mycompany.com`, and vice-versa.
cause it to redirect the client to node `A`'s
[`api_addr`](/docs/configuration#api_addr) at `https://a.vault.mycompany.com`,
and vice-versa.

### Behind Load Balancers

Expand All @@ -115,46 +126,59 @@ case, the Vault servers should actually be set up as described in the above
section, since for redirection purposes the clients have direct access.

However, if the only access to the Vault servers is via the load balancer, the
`api_addr` on each node should be the same: the address of the load
balancer. Clients that reach a standby node will be redirected back to the load
balancer; at that point hopefully the load balancer's configuration will have
been updated to know the address of the current leader. This can cause a
redirect loop and as such is not a recommended setup when it can be avoided.
[`api_addr`](/docs/configuration#api_addr) on each node should be the same: the
address of the load balancer. Clients that reach a standby node will be
redirected back to the load balancer; at that point hopefully the load
balancer's configuration will have been updated to know the address of the
current leader. This can cause a redirect loop and as such is not a recommended
setup when it can be avoided.

### Per-Node Cluster Listener Addresses

Each `listener` block in Vault's configuration file contains an `address` value
on which Vault listens for requests. Similarly, each `listener` block can
contain a `cluster_address` on which Vault listens for server-to-server cluster
requests. If this value is not set, its IP address will be automatically set to
same as the `address` value, and its port will be automatically set to the same
as the `address` value plus one (so by default, port `8201`).
Each [`listener`](/docs/configuration/listener) block in Vault's configuration
file contains an [`address`](/docs/configuration/listener/tcp#address) value on
which Vault listens for requests. Similarly, each
[`listener`](/docs/configuration/listener) block can contain a
[`cluster_address`](/docs/configuration/listener/tcp#cluster_address) on which
Vault listens for server-to-server cluster requests. If this value is not set,
its IP address will be automatically set to same as the
[`address`](/docs/configuration/listener/tcp#address) value, and its port will
be automatically set to the same as the
[`address`](/docs/configuration/listener/tcp#address) value plus one (so by
default, port `8201`).

Note that _only_ active nodes have active listeners. When a node becomes active
it will start cluster listeners, and when it becomes standby it will stop them.

### Per-Node Cluster Address

Similar to the `api_addr`, `cluster_addr` is the value that each node, if
active, should advertise to the standbys to use for server-to-server
Similar to the [`api_addr`](/docs/configuration#api_addr),
[`cluster_addr`](/docs/configuration#cluster_addr) is the value that each node,
if active, should advertise to the standbys to use for server-to-server
communications, and lives as a top-level value in the configuration file. On
each node, this should be set to a host name or IP address that a standby can
use to reach one of that node's `cluster_address` values set in the `listener`
blocks, including port. (Note that this will always be forced to `https` since
only TLS connections are used between servers.)
use to reach one of that node's
[`cluster_address`](/docs/configuration#cluster_address) values set in the
[`listener`](/docs/configuration/listener) blocks, including port. (Note that
this will always be forced to `https` since only TLS connections are used
between servers.)

This value can also be specified by the `VAULT_CLUSTER_ADDR` environment
variable, which takes precedence.

## Storage Support

Currently there are several storage backends that support high availability
mode, including Consul, ZooKeeper and etcd. These may change over time, and the
[configuration page](/docs/configuration) should be referenced.
mode, including [Consul](/docs/storage/consul),
[ZooKeeper](/docs/storage/zookeeper) and [etcd](/docs/storage/etcd). These may
change over time, and the [configuration page](/docs/configuration) should be
referenced.

The Consul backend is the recommended HA backend, as it is used in production
The [Consul backend](/docs/storage/consul) is the recommended HA backend, as it is used in production
by HashiCorp and its customers with commercial support.

If you're interested in implementing another backend or adding HA support to
another backend, we'd love your contributions. Adding HA support requires
implementing the `physical.HABackend` interface for the storage backend.
implementing the
[`physical.HABackend`](https://pkg.go.dev/github.com/hashicorp/vault/sdk/physical#HABackend)
interface for the storage backend.

0 comments on commit 625dbd8

Please sign in to comment.