When to deploy Easegress as a cluster?
- Your traffic is larger than one machine can handle
- You want to High-availability and minimize the service downtime
- You want to minimize latency on service peaks
It is easy to start multiple Easegress instances to form an Easegress cluster. This tutorial provides instructions on how to create a stand-alone Easegress cluster by starting multiple Easegress instances.
The following prerequisites are required for a successful deployment of the Easegress cluster.
- latest
easegress-serverandegctlbinaries (runmakein the root of the repository) - successful creation of an Easegress pipeline (like the Hello World example in the README.md of the repository or any other chapter in doc/cookbook)
- few machines that are in the same network or otherwise accessible or Docker or other container technology. If you only have one machine, then you could use localhost as the host and modify the ports in the example.
The goal of this tutorial is to have following infrastructure running Easegress:
- 4 machines connected
- each running Easegress instance
- 3 of Easegress instances have cluster role primary and one secondary
The difference between primary and secondary cluster roles is that primary persists the cluster state to disk, while secondary does not. The number of secondary Easegress instances can scale up and down, but the number of primary instances should be fixed.
Let's start by creating three Easegress instances with primary role. Add nodes private IPs to following environment variables:
export HOST1=<host1-IP>
export HOST2=<host2-IP>
export HOST3=<host3-IP>
export CLUSTER=machine-1=http://$HOST1:2380,machine-2=http://$HOST2:2380,machine-3=http://$HOST3:2380CLUSTER environment variable now contains IP addresses of each member in the cluster. It will be the same for all members.
Set the environment variables to each machine. Start the first instance at the first machine
easegress-server \
--cluster-name "multi-node-cluster" \
--cluster-role "primary" \
--name "machine-1" \
--api-addr $HOST1:2381 \
--initial-advertise-peer-urls http://$HOST1:2380 \
--listen-peer-urls http://$HOST1:2380 \
--listen-client-urls http://$HOST1:2379 \
--advertise-client-urls http://$HOST1:2379 \
--initial-cluster $CLUSTERHere we define the basic information, like the name of the instance and the name of the cluster. Arguments initial-advertise-peer-urls,listen-peer-urls, listen-client-urls and advertise-client-urls are for communication with other peers (other primary cluster members). You can read more about them at the end of this tutorial, but for now, it's enough to notice that hostname for machine-1 is $HOST1, which is the IP address of this machine.
Then start the second instance at machine 2
easegress-server \
--cluster-name "multi-node-cluster" \
--cluster-role "primary" \
--name "machine-2" \
--initial-advertise-peer-urls http://$HOST2:2380 \
--listen-peer-urls http://$HOST2:2380 \
--listen-client-urls http://$HOST2:2379 \
--advertise-client-urls http://$HOST2:2379 \
--initial-cluster $CLUSTERand the last machine 3.
easegress-server \
--cluster-name "multi-node-cluster" \
--cluster-role "primary" \
--name "machine-3" \
--initial-advertise-peer-urls http://$HOST3:2380 \
--listen-peer-urls http://$HOST3:2380 \
--listen-client-urls http://$HOST3:2379 \
--advertise-client-urls http://$HOST3:2379 \
--initial-cluster $CLUSTERNow you can list cluster members
egctl --server $HOST1:2381 member list | grep " name"should print
name: machine-1
name: machine-2
name: machine-3Let's add one more node with a secondary cluster role this time.
# on machine 4
easegress-server \
--cluster-name "multi-node-cluster" \
--cluster-role "secondary" \
--name "machine-4" \
--primary-listen-peer-urls http://$HOST1:2380 \
--state-flag "existing"Here primary-listen-peer-urls tell, where to find a primary cluster members and state-flag with value "existing" means that this cluster was already created.
We can now see also the 4th instance:
egctl --server $HOST1:2381 member list | grep " name"prints to output:
name: machine-1
name: machine-2
name: machine-3
name: machine-4Congratulations, you now have your Easegress instances running! You can now start applying resources to Easegress, like pipeline or workflow for example.
You can also keep reading this tutorial to know more about YAML configuration of Easegress cluster instances or configuration tips.
The examples above use the easegress-server's command-line flags, but often it is more convenient to define server parameters in a YAML configuration file. For example, store the following YAML to each host machine and change the host addresses accordingly.
# create one yaml file for each host
name: machine-1 # machine-2, machine-3
cluster-name: cluster-test
cluster-role: primary
api-addr: localhost:2381
data-dir: ./data
wal-dir: ""
cpu-profile-file:
memory-profile-file:
log-dir: ./log
debug: false
cluster:
listen-peer-urls: # change CURRENT-HOST to current host
- http://<CURRENT-HOST>:2380
listen-client-urls:
- http://<CURRENT-HOST>:2379
advertise-client-urls:
- http://<CURRENT-HOST>:2379
initial-advertise-peer-urls:
- http://<CURRENT-HOST>:2380
initial-cluster: # initial-cluster is same for every host
- machine-1: http://<HOST-1>:2380
- machine-2: http://<HOST-2>:2380
- machine-3: http://<HOST-3>:2380Then apply these values on each machine, using config-file command line argument:
easegress-server --config-file config.yaml.
The configuration file for adding a new secondary node looks like following:
name: machine-4
cluster-name: cluster-test
cluster-role: secondary
data-dir: ./data
wal-dir: ""
cpu-profile-file:
memory-profile-file:
log-dir: ./log
debug: false
cluster:
primary-listen-peer-urls:
- http://$HOST1:2380What is a good size for the cluster?
It is a good practice to choose an odd number (1,3,5,7,9) of primary nodes, to tolerate failures of primary nodes. This way the cluster can stay in a healthy state, even if the network partitions. With an even number of primary nodes, the cluster can be divided into two groups of equal size due to network partition. Then neither of the sub-clusters have the majority required for consensus. However, with the odd number of primary nodes, the cluster cannot be divided into two groups of equal size and this problem cannot occur.
For the secondary nodes, there are no constraints for the number of nodes. Secondary nodes do not participate consensus vote of the cluster, so their failure does not affect the cluster health. Adding more (secondary) nodes does still increase the communication between nodes.
Can a number of primary members scale up?
Please note that it is not recommended to add additional node with primary cluster role, but primary nodes should be started at cluster start up. When scaling up the cluster, it is recommended to add and remove secondary cluster members.
What are advertise-peer-urls and listen-client-urls?
advertise-client-urls, listen-client-urls, listen-peer-urls and listen-client-urls are arguments necessary for primary cluster members to communicate with other peers (members).
| argument | description |
|---|---|
| advertise-client-urls | client URLs member advertises to the rest of the cluster |
| listen-client-urls | URLs that member listens for client traffic |
| listen-peer-urls | URLs that member listens for peer (other primary members) traffic |
| initial-advertise-peer-urls | peer (other primary members) URLs member advertises to the rest of the cluster |
These arguments are used for etcd server and client configuration. You can read more about them in etcd documentation.
Why primary member needs more arguments than secondary member ?
Here's a drawing that illustrates the difference.
Primary members need to synchronize with peers (other primary members). Meanwhile secondary can read and update the state through one primary member.
How are etcd server and Easegress primary member related?
Easegress uses etcd distributed key-value store to synchronize the cluster state. The primary and secondary cluster roles have the following relation with etcd:
| Easegress cluster role | primary | secondary |
|---|---|---|
| etcd term | server | client |
Primary member uses etcd server for cluster communication, while secondary member uses etcd client for this.