This manual will show how to install and configure a Redis-Cluster on Ubuntu. test
Prerequisits
Redis has to be installed on a Linux OS (Ubuntu, Redhat, Centos, etc.).
To install Redis, at least 3 instances are needed. The instances can be distributed across different servers, or concentrated on one (best practice will be shown in the examples below).
If the servers for Redis are in different network-zones, the servers must be able to communicate with eachother.
The following Firewall-Ports must be open:
6379
16384
16379
7001 (Cluster)
7002 (Cluster)
7003 (Cluster)
7004 (Cluster)
7006 (FCluster)
Example
In our example we got 3 virtual Ubuntu-Servers.
Redis HA 1 (192.168.17.134)
Redis HA 2 (192.168.17.135)
Redis HA 3 (192.168.17.136)
On Redis HA1 is a Master Instance 7001 and a Slave Instance 7004
On Redis HA2 is a Master Instance 7002 and a Slave Instance 7005
On Redis HA3 is a Master Instance 7003 and a Slave Instance 7006
For the Master 7001 the Slave is 7006
For the Master 7002 the Slave is 7006
For the Master 7003 the Slave is 7004
Installation
General installation for all 3 servers:
Switch to root user:
sudo su
Create a Folder in /etc/redis in which Redis will be installed:
mkdir /etc/redis
Download Redis and unzipp the package
https://download.redis.io/releases/redis-6.2.6.tar.gz can also be downloaded on windows and copied to your linux instance.
wget https://download.redis.io/releases/redis-6.2.6.tar.gz tar xzf redis-6.2.6.tar.gz
This will create the folder /Files in /etc/redis.
Navigate with ‘cd’ to the folder /redis-6.2.6
Redis needs gcc, make and tcl. Those can be installed with the following command:
sudo apt-get install -y tcl tk sudo apt install gcc sudo apt-get install make
Check if the local firewall is active:
sudo ufw status
Open the needed Ports on the firewall:
sudo ufw allow 6379 sudo ufw allow 16384 sudo ufw allow 16379 sudo ufw allow 7001 sudo ufw allow 7002 sudo ufw allow 7003 sudo ufw allow 7004 sudo ufw allow 7005 sudo ufw allow 7006
Since we installed ‘make’, we have to exexute the File in etc/redis/redis-6.2.6/makefile
cd /etc/redis/redis-6.2.6/ sudo make test
Configuration VM 1
In our example we have multiple redis instanced on one server. This means we need multiple config-files on the server as well.
We will create on VM01 the following config-Files →node1.conf / node4.conf . Those will be used to start redis.
Create config-file 1 & 4
If Vim is not installed yet, use the command “apt install vim” to install it.
Create the config-Files in the folder /etc/redis - to navigate there, use the following command:
cd /etc/redis
sudo vim node1.conf
Copy the following text to the config file:
port 7001 cluster-enabled yes cluster-config-file cluster-node-1.conf cluster-node-timeout 5000 appendonly yes appendfilename node-1.aof dbfilename dump-1.rdb
vim: Save file ctrl+c | wq!
vim: Close file without saving ctrl+c | q!
vim: Search in file esc /search
vim: Edit File: i
sudo vim node4.conf
Copy the following text to the config file:
port 7004 cluster-enabled yes cluster-config-file cluster-node-4.conf cluster-node-timeout 5000 appendonly yes appendfilename node-4.aof dbfilename dump-4.rdb
Configuration VM 2
In our example we have multiple redis instanced on one server. This means we need multiple config-files on the server as well.
We will create on VM02 the following config-Files →node2.conf / node5.conf . Those will be used to start redis.
Create config-file 2 & 5
If Vim is not installed yet, use the command “apt install vim” to install it.
Create the config-Files in the folder /etc/redis - to navigate there, use the following command:
cd /etc/redis sudo vim node2.conf
Copy the following text to the config file:
port 7002 cluster-enabled yes cluster-config-file cluster-node-2.conf cluster-node-timeout 5000 appendonly yes appendfilename node-2.aof dbfilename dump-2.rdb
vim: Save file ctrl+c | wq!
vim: Close file without saving ctrl+c | q!
vim: Search in file esc /search
vim: Edit File: i
sudo vim node5.conf
Copy the following text to the config file:
port 7005 cluster-enabled yes cluster-config-file cluster-node-5.conf cluster-node-timeout 5000 appendonly yes appendfilename node-5.aof dbfilename dump-5.rdb
Configuration VM 3
In our example we have multiple redis instanced on one server. This means we need multiple config-files on the server as well.
We will create on VM03 the following config-Files →node3.conf / node6.conf . Those will be used to start redis.
Create config-file 3 & 6
If Vim is not installed yet, use the command “apt install vim” to install it.
Create the config-Files in the folder /etc/redis - to navigate there, use the following command:
cd /etc/redis sudo vim node3.conf
Copy the following text:
port 7003 cluster-enabled yes cluster-config-file cluster-node-3.conf cluster-node-timeout 5000 appendonly yes appendfilename node-3.aof dbfilename dump-3.rdb
vim: Save file ctrl+c | wq!
vim: Close file without saving ctrl+c | q!
vim: Search in file esc /search
vim: Edit File: i
sudo vim node6.conf
Copy the following text:
port 7006 cluster-enabled yes cluster-config-file cluster-node-6.conf cluster-node-timeout 5000 appendonly yes appendfilename node-6.aof dbfilename dump-6.rdb
Start Redis
We can start Redis now on each VM through the config-files we created.
When a server shuts down, the application does not automatically restart. So you have to start the redis instances again!
To do this, navigate to the folder /etc/redis and execute the following commands:
cd /etc/redis
Server | Instanz | Befehl |
|---|---|---|
HA1 (192.168.17.134) | Node 01 - Port 7001 Node 04 - Port 7004 | ./redis-6.2.6/src/redis-server node1.conf & ./redis-6.2.6/src/redis-server node4.conf & |
HA 2 (192.168.17.135) | Node 02 - Port 7002 Node 05 - Port 7005 | ./redis-6.2.6/src/redis-server node2.conf & ./redis-6.2.6/src/redis-server node5.conf & |
HA 3 (192.168.17.136) | Node 03 - Port 7003 Node 06 - Port 7006 | ./redis-6.2.6/src/redis-server node3.conf & ./redis-6.2.6/src/redis-server node6.conf & |
Connect Redis
We can now connect on each instace with the Redis-CLI
Server | Instanz | Befehl |
|---|---|---|
HA1 (192.168.17.134) | Node 01 - Port 7001 Node 04 - Port 7004 | ./redis-6.2.6/src/redis-cli -p 7001 ./redis-6.2.6/src/redis-cli -p 7004 |
HA 2 (192.168.17.135) | Node 02 - Port 7002 Node 05 - Port 7005 | ./redis-6.2.6/src/redis-cli -p 7002 ./redis-6.2.6/src/redis-cli -p 7005 |
HA 3 (192.168.17.136) | Node 03 - Port 7003 Node 06 - Port 7006 | ./redis-6.2.6/src/redis-cli -p 7003 ./redis-6.2.6/src/redis-cli -p 7006 |
Configure Redis Cluster
Since the instaces are local at the moment, we need to connect them to the cluster.
Navigate to /etc/redis
cd /etc/redis
./redis-6.2.6/src/redis-cli --cluster + all instances. The second instance will always be the slave. --cluster-replicas shows how many slaves will be created for each master.
./redis-6.2.6/src/redis-cli --cluster create 192.168.17.134:7001 192.168.17.134:7004 192.168.17.135:7002 192.168.17.135:7005 192.168.17.136:7003 192.168.17.136:7006 --cluster-replicas 1
Check the config and confirm with: Y
Redis-CLI
Open Redis-CLI:
-p = Port -c = Cluster
./redis-6.2.6/src/redis-cli -c -p 7001
ping
ping
The different nodes can be pinged:
info
info
Shows all information on the cluster
cluster nodes
cluster nodes
Shows all nodes and if they are master or slave.
info replication
info replication
Shows if the server is master or slave. Shows the depending slaves.
Failover
We will terminate Master 7003 for this example. This will prompt the Slave 7004 to become the new master:
Shutdown the instance
./redis-6.2.6/src/redis-cli -c -p 7003 shutdown
On the masters 7001 and 7002 we will get the following message:
2074:S 26 Apr 2022 13:56:22.765 # Error condition on socket for SYNC: Connection refused 2074:S 26 Apr 2022 13:56:23.773 * Connecting to MASTER 192.168.17.136:7003 2074:S 26 Apr 2022 13:56:23.773 * MASTER <-> REPLICA sync started 2074:S 26 Apr 2022 13:56:23.773 # Error condition on socket for SYNC: Connection refused 3904:M 26 Apr 2022 13:56:24.099 * FAIL message received from 9b989e13f6e83d661c072853f48d89d1a995f69c about 88ff5fc7513281e33e5397175cba7dbec2d01265 3904:M 26 Apr 2022 13:56:24.099 # Cluster state changed: fail 2074:S 26 Apr 2022 13:56:24.100 * FAIL message received from 9b989e13f6e83d661c072853f48d89d1a995f69c about 88ff5fc7513281e33e5397175cba7dbec2d01265 2074:S 26 Apr 2022 13:56:24.100 # Cluster state changed: fail 2074:S 26 Apr 2022 13:56:24.177 # Start of election delayed for 711 milliseconds (rank #0, offset 589126). 2074:S 26 Apr 2022 13:56:24.781 * Connecting to MASTER 192.168.17.136:7003 2074:S 26 Apr 2022 13:56:24.781 * MASTER <-> REPLICA sync started 2074:S 26 Apr 2022 13:56:24.781 # Error condition on socket for SYNC: Connection refused 2074:S 26 Apr 2022 13:56:24.983 # Starting a failover election for epoch 28. 3904:M 26 Apr 2022 13:56:24.984 # Failover auth granted to cdfafbc0721ba7774a6e9412e51e7369bd150c1a for epoch 28 2074:S 26 Apr 2022 13:56:25.006 # Failover election won: I'm the new master. 2074:S 26 Apr 2022 13:56:25.006 # configEpoch set to 28 after successful failover 2074:M 26 Apr 2022 13:56:25.006 * Discarding previously cached master state. 2074:M 26 Apr 2022 13:56:25.006 # Setting secondary replication ID to ddd94fd4c80c068c06c03fc5113ce9f33cbbe780, valid up to offset: 589127. New replication ID is 897d8ffc57482c31a6d37db92136073b713e813e 2074:M 26 Apr 2022 13:56:25.006 # Cluster state changed: ok 3904:M 26 Apr 2022 13:56:25.024 # Cluster state changed: ok
We can now check with the Redis-CLI. With the command “Cluster nodes”, we can see, Slave 7004 is now a Master.
We can also see that Master 7003 can’t be reached.
If we start the Slave 7004, who is now a Master, the server will still be described as Slave.
./redis-6.2.6/src/redis-server node3.conf &
Message after restart:
17830:S 26 Apr 2022 14:05:46.962 * MASTER <-> REPLICA sync: receiving 193 bytes from master to disk 17830:S 26 Apr 2022 14:05:46.962 * MASTER <-> REPLICA sync: Flushing old data 17830:S 26 Apr 2022 14:05:46.962 * MASTER <-> REPLICA sync: Loading DB in memory 17830:S 26 Apr 2022 14:05:46.963 * Loading RDB produced by version 6.2.6 17830:S 26 Apr 2022 14:05:46.963 * RDB age 0 seconds 17830:S 26 Apr 2022 14:05:46.963 * RDB memory usage when created 2.54 Mb 17830:S 26 Apr 2022 14:05:46.963 # Done loading RDB, keys loaded: 2, keys expired: 0. 17830:S 26 Apr 2022 14:05:46.963 * MASTER <-> REPLICA sync: Finished with success 17830:S 26 Apr 2022 14:05:46.963 * Background append only file rewriting started by pid 17836 17830:S 26 Apr 2022 14:05:47.001 * AOF rewrite child asks to stop sending diffs. 17836:C 26 Apr 2022 14:05:47.001 * Parent agreed to stop sending diffs. Finalizing AOF... 17836:C 26 Apr 2022 14:05:47.001 * Concatenating 0.00 MB of AOF diff received from parent. 17836:C 26 Apr 2022 14:05:47.001 * SYNC append only file rewrite performed 17836:C 26 Apr 2022 14:05:47.001 * AOF rewrite: 0 MB of memory used by copy-on-write 17830:S 26 Apr 2022 14:05:47.075 * Background AOF rewrite terminated with success 17830:S 26 Apr 2022 14:05:47.075 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB) 17830:S 26 Apr 2022 14:05:47.075 * Background AOF rewrite finished successfully
If you want the 7003 as Master again, use the following command:
./redis-6.2.6/src/redis-cli -c -p 7003 cluster failover
Message:
127.0.0.1:7003> 17830:S 26 Apr 2022 14:07:57.685 # Received replication offset for paused master manual failover: 589308 17830:S 26 Apr 2022 14:07:57.685 # All master replication stream processed, manual failover can start. 7830:S 26 Apr 2022 14:07:57.685 # Start of election delayed for 0 milliseconds (rank #0, offset 589308). 17830:S 26 Apr 2022 14:07:57.685 # Starting a failover election for epoch 29. 17830:S 26 Apr 2022 14:07:57.686 # Currently unable to failover: Waiting for votes, but majority still not reached. 17830:S 26 Apr 2022 14:07:57.686 # Failover election won: I'm the new master. 17830:S 26 Apr 2022 14:07:57.686 # configEpoch set to 29 after successful failover 17830:M 26 Apr 2022 14:07:57.686 # Connection with master lost. 17830:M 26 Apr 2022 14:07:57.686 * Caching the disconnected master state. 17830:M 26 Apr 2022 14:07:57.686 * Discarding previously cached master state. 17830:M 26 Apr 2022 14:07:57.686 # Setting secondary replication ID to 897d8ffc57482c31a6d37db92136073b713e813e, valid up to offset: 589309. New replication ID is 17e90a8ddf3d4fd645d4ec162267629a4109ea9f
Check the cluster nodes:
