This manual will show how to install and configure a Redis-Cluster on Ubuntu.

...

Introduction

Redis is an in-memory data storage that is used by the CoreOne Suite. For supported versions see: Next-Gen Workflows - Redis

Step 1 - Check your prerequisits

• Redis has to be installed on a Linux OS (Ubuntu, Redhat, Centos, etc.).

• To install Redis, at least 3 instances are needed. The instances can be distributed across different servers, or concentrated on one (best practice will be shown in the examples below).

• If the servers for Redis are in different network-zones, the servers must be able to communicate with eachother.

• The following Firewall-Ports must be open:

  • 6379

  • 16384

  • 16379

  • 7001 (Cluster)

  • 7002 (Cluster)

  • 7003 (Cluster)

  • 7004 (Cluster)

  • 7006 (FClusterCluster)

Example

In our example we got 3 virtual Ubuntu-Servers.

Redis HA 1 (192.168.17.134)
Redis HA 2 (192.168.17.135)
Redis HA 3 (192.168.17.136)

On Redis HA1 is a Master Instance 7001 and a Slave Instance 7004
On Redis HA2 is a Master Instance 7002 and a Slave Instance 7005
On Redis HA3 is a Master Instance 7003 and a Slave Instance 7006

For the Master 7001 the Slave is 7006
For the Master 7002 the Slave is 7006
For the Master 7003 the Slave is 7004

...

Step 2 - Installation

General installation for all 3 servers:

Switch to root user:

sudo su

...

cd /etc/redis/redis-6.2.6/
sudo make test

Step 3 - Configuration VM 1

Da wir wie in unserem Beispiel mehrere Redis-Instanzen pro Server haben. Brauchen wir mehrere konfig-Files pro Server.
Das heisst wir erstellen auf der VM01 folgende In our example we have multiple redis instanced on one server. This means we need multiple config-files on the server as well.

We will create on VM01 the following config-Files →node1.conf / node4.conf . Mit welchen wir anschliessend Redis starten. Config-File Those will be used to start redis.

Create config-file 1 & 4 erstellen

...

Create the config-Files im in the folder /etc/redis - sprich noch in das Verzeichnis navigierento navigate there, use the following command:

cd /etc/redis

sudo vim node1.conf

Folgender Text kopieren;Copy the following text to the config file:

port 7001
cluster-enabled yes
cluster-config-file cluster-node-1.conf
cluster-node-timeout 5000
appendonly yes
appendfilename node-1.aof
dbfilename dump-1.rdb

...

...

...

...

...

sudo vim node4.conf

Folgender Text kopieren;Copy the following text to the config file:

port 7004
cluster-enabled yes
cluster-config-file cluster-node-4.conf
cluster-node-timeout 5000
appendonly yes
appendfilename node-4.aof
dbfilename dump-4.rdb

...

Step 4 - Configuration VM 2

Da wir wie in unserem Beispiel mehrere Redis-Instanzen pro Server haben. Brauchen wir mehrere konfig-Files pro Server.
Das heisst wir erstellen auf der VM02 folgende In our example we have multiple redis instanced on one server. This means we need multiple config-files on the server as well.

We will create on VM02 the following config-Files →node2.conf / node5.conf . Mit welchen wir anschliessend Redis starten. Config-File Those will be used to start redis.

Create config-file 2 & 5 erstellen

...

Create the config-Files im in the folder /etc/redis - sprich noch in das Verzeichnis navigierento navigate there, use the following command:

cd /etc/redis
sudo vim node2.conf

Folgender Text kopieren;Copy the following text to the config file:

port 7002
cluster-enabled yes
cluster-config-file cluster-node-2.conf
cluster-node-timeout 5000
appendonly yes
appendfilename node-2.aof
dbfilename dump-2.rdb

...

...

...

...

...

sudo vim node5.conf

Folgender Text kopieren;Copy the following text to the config file:

port 7005
cluster-enabled yes
cluster-config-file cluster-node-5.conf
cluster-node-timeout 5000
appendonly yes
appendfilename node-5.aof
dbfilename dump-5.rdb

...

Step 5 - Configuration VM 3

Da wir wie in unserem Beispiel mehrere Redis-Instanzen pro Server haben. Brauchen wir mehrere konfig-Files pro Server.
Das heisst wir erstellen auf der VM03 folgende In our example we have multiple redis instanced on one server. This means we need multiple config-files on the server as well.

We will create on VM03 the following config-Files →node3.conf / node6.conf . Mit welchen wir anschliessend Redis starten. Config-File Those will be used to start redis.

Create config-file 3 & 6 erstellen

...

Create the config-Files im in the folder /etc/redis - sprich noch in das Verzeichnis navigierento navigate there, use the following command:

cd /etc/redis
sudo vim node3.conf

Folgender Text kopieren;Copy the following text:

port 7003
cluster-enabled yes
cluster-config-file cluster-node-3.conf
cluster-node-timeout 5000
appendonly yes
appendfilename node-3.aof
dbfilename dump-3.rdb

...

...

...

...

...

sudo vim node6.conf

Folgender Text kopieren;Copy the following text:

port 7006
cluster-enabled yes
cluster-config-file cluster-node-6.conf
cluster-node-timeout 5000
appendonly yes
appendfilename node-6.aof
dbfilename dump-6.rdb

Step 6 - Start Redis

...

Auf jeder VM kann nun mit folgeden Befehl Redis mit einem Config-File gestartet werden.

...

We can start Redis now on each VM through the config-files we created.

Info
When a server shuts down, the application does not automatically restart. So you have to start the redis instances again!

To do this, navigate to the folder /etc/redis and execute the following commands:

Code Block
cd /etc/redis

Server	Instanz	Befehl
HA1 (192.168.17.134)	Node 01 - Port 7001 Node 04 - Port 7004	Code Block ./redis-6.2.6/src/redis-server node1.conf & Code Block ./redis-6.2.6/src/redis-server node4.conf &
HA 2 (192.168.17.135)	Node 02 - Port 7002 Node 05 - Port 7005	Code Block ./redis-6.2.6/src/redis-server node2.conf & Code Block ./redis-6.2.6/src/redis-server node5.conf &
HA 3 (192.168.17.136)	Node 03 - Port 7003 Node 06 - Port 7006	Code Block ./redis-6.2.6/src/redis-server node3.conf & Code Block ./redis-6.2.6/src/redis-server node6.conf &

Step 7 - Connect Redis

...

We can now connect on each instace with the Redis-CLI zugegriffen werden.

Server	Instanz	Befehl
HA1 (192.168.17.134)	Node 01 - Port 7001 Node 04 - Port 7004	Code Block ./redis-6.2.6/src/redis-cli -p 7001 ./redis-6.2.6/src/redis-cli -p 7004
HA 2 (192.168.17.135)	Node 02 - Port 7002 Node 05 - Port 7005	Code Block ./redis-6.2.6/src/redis-cli -p 7002 ./redis-6.2.6/src/redis-cli -p 7005
HA 3 (192.168.17.136)	Node 03 - Port 7003 Node 06 - Port 7006	Code Block ./redis-6.2.6/src/redis-cli -p 7003 ./redis-6.2.6/src/redis-cli -p 7006

Step 8 - Configure Redis Cluster

...

Da die Instanzen jedoch noch Lokal laufen, müssen diese dem Cluster hinzugefügt werden.

...

Since the instaces are local at the moment, we need to connect them to the cluster.

Navigate to /etc/redis

Code Block
cd /etc/redis

Info
./redis-6.2.6/src/redis-cli --cluster + alle Instanzen. Die zweite Instanz ist in jedem Fall der Slaveall instances. The second instance will always be the slave. --cluster-replicas gibt an wie viele slave pro Master erstellt wirdshows how many slaves will be created for each master.

Code Block
./redis-6.2.6/src/redis-cli --cluster create 192.168.17.134:7001 192.168.17.134:7004 192.168.17.135:7002 192.168.17.135:7005 192.168.17.136:7003 192.168.17.136:7006 --cluster-replicas 1

...

Überprüfen und mit Y bestätigen. Check the config and confirm with: Y

Step 9 - Redis-CLI

Auf die Open Redis-CLI kommt man wie folgt:

./redis-6.2.6/src/redis-cli -c -p 7001

ping

ping

Können andere Nodes gepingt werden.The different nodes can be pinged:

info

info

Zeigt alle Informationen über den Cluster anShows all information on the cluster

...

cluster nodes

cluster nodes

Zeigt alle Nodes an, und ob diese Master oder Slave sindShows all nodes and if they are master or slave.

...

info replication

info replication

Zeigt an, ob der Server Master oder Slave ist und wer dessen Slave istShows if the server is master or slave. Shows the depending slaves.

...

Step 10 - Failover

Wenn wir nun als Beispiel den We will terminate Master 7003 beenden.
Müsste der Slave 7004 als neuen Master fungieren.

...

Für dieses Beispiel fahren wir den Master 7003 herunter. for this example. This will prompt the Slave 7004 to become the new master:

...

Shutdown the instance

./redis-6.2.6/src/redis-cli -c -p 7003
shutdown

...

Auf dem Master von 7001 und 7002 erhält man nun folgende Meldung:

2074:S 26 Apr 2022 13:56:22.765 # Error condition on socket for SYNC: Connection refused
2074:S 26 Apr 2022 13:56:23.773 * Connecting to MASTER 192.168.17.136:7003
2074:S 26 Apr 2022 13:56:23.773 * MASTER <-> REPLICA sync started
2074:S 26 Apr 2022 13:56:23.773 # Error condition on socket for SYNC: Connection refused
3904:M 26 Apr 2022 13:56:24.099 * FAIL message received from 9b989e13f6e83d661c072853f48d89d1a995f69c about 88ff5fc7513281e33e5397175cba7dbec2d01265
3904:M 26 Apr 2022 13:56:24.099 # Cluster state changed: fail
2074:S 26 Apr 2022 13:56:24.100 * FAIL message received from 9b989e13f6e83d661c072853f48d89d1a995f69c about 88ff5fc7513281e33e5397175cba7dbec2d01265
2074:S 26 Apr 2022 13:56:24.100 # Cluster state changed: fail
2074:S 26 Apr 2022 13:56:24.177 # Start of election delayed for 711 milliseconds (rank #0, offset 589126).
2074:S 26 Apr 2022 13:56:24.781 * Connecting to MASTER 192.168.17.136:7003
2074:S 26 Apr 2022 13:56:24.781 * MASTER <-> REPLICA sync started
2074:S 26 Apr 2022 13:56:24.781 # Error condition on socket for SYNC: Connection refused
2074:S 26 Apr 2022 13:56:24.983 # Starting a failover election for epoch 28.
3904:M 26 Apr 2022 13:56:24.984 # Failover auth granted to cdfafbc0721ba7774a6e9412e51e7369bd150c1a for epoch 28
2074:S 26 Apr 2022 13:56:25.006 # Failover election won: I'm the new master.
2074:S 26 Apr 2022 13:56:25.006 # configEpoch set to 28 after successful failover
2074:M 26 Apr 2022 13:56:25.006 * Discarding previously cached master state.
2074:M 26 Apr 2022 13:56:25.006 # Setting secondary replication ID to ddd94fd4c80c068c06c03fc5113ce9f33cbbe780, valid up to offset: 589127. New replication ID is 897d8ffc57482c31a6d37db92136073b713e813e
 2074:M 26 Apr 2022 13:56:25.006 # Cluster state changed: ok
 3904:M 26 Apr 2022 13:56:25.024 # Cluster state changed: ok

...

Wenn der ehmalige Slave 7004 nun wieder gestartet wird, ist dieser Standardmässig noch als Slave hinterlegtOn the masters 7001 and 7002 we will get the following message:

We can now check with the Redis-CLI. With the command “Cluster nodes”, we can see, Slave 7004 is now a Master.
We can also see that Master 7003 can’t be reached.

...

If we start the Slave 7004, who is now a Master, the server will still be described as Slave.

./redis-6.2.6/src/redis-server node3.conf &

Meldung nach dem AufstartenMessage after restart:

17830:S 26 Apr 2022 14:05:46.962 * MASTER <-> REPLICA sync: receiving 193 bytes from master to disk
17830:S 26 Apr 2022 14:05:46.962 * MASTER <-> REPLICA sync: Flushing old data
17830:S 26 Apr 2022 14:05:46.962 * MASTER <-> REPLICA sync: Loading DB in memory
17830:S 26 Apr 2022 14:05:46.963 * Loading RDB produced by version 6.2.6
17830:S 26 Apr 2022 14:05:46.963 * RDB age 0 seconds
17830:S 26 Apr 2022 14:05:46.963 * RDB memory usage when created 2.54 Mb
17830:S 26 Apr 2022 14:05:46.963 # Done loading RDB, keys loaded: 2, keys expired: 0.
17830:S 26 Apr 2022 14:05:46.963 * MASTER <-> REPLICA sync: Finished with success
17830:S 26 Apr 2022 14:05:46.963 * Background append only file rewriting started by pid 17836
17830:S 26 Apr 2022 14:05:47.001 * AOF rewrite child asks to stop sending diffs.
17836:C 26 Apr 2022 14:05:47.001 * Parent agreed to stop sending diffs. Finalizing AOF...
17836:C 26 Apr 2022 14:05:47.001 * Concatenating 0.00 MB of AOF diff received from parent.
17836:C 26 Apr 2022 14:05:47.001 * SYNC append only file rewrite performed
17836:C 26 Apr 2022 14:05:47.001 * AOF rewrite: 0 MB of memory used by copy-on-write
17830:S 26 Apr 2022 14:05:47.075 * Background AOF rewrite terminated with success
17830:S 26 Apr 2022 14:05:47.075 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
17830:S 26 Apr 2022 14:05:47.075 * Background AOF rewrite finished successfully

Wenn man den 7003 wieder als Master deklarieren möchte, kann man dies mitIf you want the 7003 as Master again, use the following command:

./redis-6.2.6/src/redis-cli -c -p 7003
cluster failover

MeldungMessage:

127.0.0.1:7003> 17830:S 26 Apr 2022 14:07:57.685 # Received replication offset for paused master manual failover: 589308
17830:S 26 Apr 2022 14:07:57.685 # All master replication stream processed, manual failover can start.
7830:S 26 Apr 2022 14:07:57.685 # Start of election delayed for 0 milliseconds (rank #0, offset 589308).
17830:S 26 Apr 2022 14:07:57.685 # Starting a failover election for epoch 29.
17830:S 26 Apr 2022 14:07:57.686 # Currently unable to failover: Waiting for votes, but majority still not reached.
17830:S 26 Apr 2022 14:07:57.686 # Failover election won: I'm the new master.
17830:S 26 Apr 2022 14:07:57.686 # configEpoch set to 29 after successful failover
17830:M 26 Apr 2022 14:07:57.686 # Connection with master lost.
17830:M 26 Apr 2022 14:07:57.686 * Caching the disconnected master state.
17830:M 26 Apr 2022 14:07:57.686 * Discarding previously cached master state.
17830:M 26 Apr 2022 14:07:57.686 # Setting secondary replication ID to 897d8ffc57482c31a6d37db92136073b713e813e, valid up to offset: 589309. New replication ID is 17e90a8ddf3d4fd645d4ec162267629a4109ea9f

Erneute überprüfung über Cluster nodesCheck the cluster nodes:

...