Redis哨兵模式快速安装以及测试

Fri, Aug 2, 2019

0 基础方面

基础方面可以参考文档： https://www.cnblogs.com/kevingrace/p/9004460.html

1 机器

master: k8s-01 slave1: k8s-02 slave2: k8s-03

2 安装(所有节点执行)

以下以脚本方式安装 install_redis.sh

REDIS_PATH=/root
REDIS_TAG=redis-5.0.5.tar.gz
REDIS_TAGNAME=redis-5.0.5
REDIS_INSTALL_PATH=/root/redis_sentinel

function install_redis () {
#################################################################################################
        cd $REDIS_PATH
        echo " $REDIS_TAG"
        if [ ! -f "$REDIS_TAG" ]; then
           echo download
           wget http://download.redis.io/releases/$REDIS_TAG
        fi
        mkdir -p $REDIS_INSTALL_PATH
        echo "=====end mkdir -p $REDIS_INSTALL_PATH======"
        tar -zxf $REDIS_PATH/$REDIS_TAG -C $REDIS_INSTALL_PATH
        echo "=====end tar -zxf $REDIS_PATH/$REDIS_TAG -C $REDIS_INSTALL_PAT ====="
        cd $REDIS_INSTALL_PATH/$REDIS_TAGNAME
        make PREFIX=/usr/local/redis install
        echo "=====end make PREFIX=/usr/local/redis install====="
        mkdir -p /usr/local/redis/{etc,var}
 #################################################################################################
}

install_redis

在master节点上安装后，将脚本(或者加上安装包)复制到slave节点上，并在slave节点上执行脚本

3 redis启停脚本(所有节点执行)

cat > /etc/init.d/redis-server << EOF
#!/bin/bash
#
# redis - this script starts and stops the redis-server daemon
#
# chkconfig:   - 85 15
# description:  Redis is a persistent key-value database
# processname: redis-server
# config:      /usr/local/redis/etc/redis.conf
# config:      /etc/sysconfig/redis
# pidfile:     /usr/local/redis/var/redis-server.pid
  
# Source function library.
. /etc/rc.d/init.d/functions
  
# Source networking configuration.
. /etc/sysconfig/network
  
# Check that networking is up.
[ "\$NETWORKING" = "no" ] && exit 0
  
redis="/usr/local/redis/bin/redis-server"
prog=\$(basename \$redis)
  
REDIS_CONF_FILE="/usr/local/redis/etc/redis.conf"
  
[ -f /etc/sysconfig/redis ] && . /etc/sysconfig/redis
  
lockfile=/var/lock/subsys/redis-server
  
start() {
    [ -x \$redis ] || exit 5
    [ -f \$REDIS_CONF_FILE ] || exit 6
    echo -n \$"Starting \$prog: "
    daemon \$redis \$REDIS_CONF_FILE
    retval=\$?
    echo
    [ \$retval -eq 0 ] && touch \$lockfile
    return \$retval
}
  
stop() {
    echo -n \$"Stopping \$prog: "
    killproc \$prog
    retval=\$?
    echo
    [ \$retval -eq 0 ] && rm -f \$lockfile
    return \$retval
}
  
restart() {
    stop
    start
}
  
reload() {
    echo -n \$"Reloading \$prog: "
    killproc \$redis -HUP
    RETVAL=\$?
    echo
}
  
force_reload() {
    restart
}
  
rh_status() {
    status \$prog
}
  
rh_status_q() {
    rh_status >/dev/null 2>&1
}
  
case "\$1" in
    start)
        rh_status_q && exit 0
        \$1
        ;;
    stop)
        rh_status_q || exit 0
        \$1
        ;;
    restart)
        \$1
        ;;
    reload)
        rh_status_q || exit 7
        \$1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo \$"Usage: \$0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
EOF

chmod 755 /etc/init.d/redis-server

4 redis-sentinel启停脚本(所有节点执行)

cat > /etc/init.d/redis-sentinel << EOF
#!/bin/bash
#
# redis-sentinel - this script starts and stops the redis-server sentinel daemon
#
# chkconfig:   - 85 15
# description:  Redis sentinel
# processname: redis-server
# config:      /usr/local/redis/etc/sentinel.conf
# config:      /etc/sysconfig/redis
# pidfile:     /usr/local/redis/var/redis-sentinel.pid
  
# Source function library.
. /etc/rc.d/init.d/functions
  
# Source networking configuration.
. /etc/sysconfig/network
  
# Check that networking is up.
[ "\$NETWORKING" = "no" ] && exit 0
  
redis="/usr/local/redis/bin/redis-sentinel"
prog=\$(basename \$redis)
  
REDIS_CONF_FILE="/usr/local/redis/etc/sentinel.conf"
  
[ -f /etc/sysconfig/redis ] && . /etc/sysconfig/redis
  
lockfile=/var/lock/subsys/redis-sentinel
  
start() {
    [ -x \$redis ] || exit 5
    [ -f \$REDIS_CONF_FILE ] || exit 6
    echo -n \$"Starting \$prog: "
    daemon \$redis \$REDIS_CONF_FILE --sentinel
    retval=\$?
    echo
    [ \$retval -eq 0 ] && touch \$lockfile
    return \$retval
}
  
stop() {
    echo -n \$"Stopping \$prog: "
    killproc \$prog
    retval=\$?
    echo
    [ \$retval -eq 0 ] && rm -f \$lockfile
    return \$retval
}
  
restart() {
    stop
    start
}
  
reload() {
    echo -n \$"Reloading \$prog: "
    killproc \$redis -HUP
    RETVAL=\$?
    echo
}
  
force_reload() {
    restart
}
  
rh_status() {
    status \$prog
}
  
rh_status_q() {
    rh_status >/dev/null 2>&1
}
  
case "\$1" in
    start)
        rh_status_q && exit 0
        \$1
        ;;
    stop)
        rh_status_q || exit 0
        \$1
        ;;
    restart)
        \$1
        ;;
    reload)
        rh_status_q || exit 7
        \$1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo \$"Usage: \$0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
EOF

chmod 755 /etc/init.d/redis-sentinel

5 主节点redis.conf配置

cat > /usr/local/redis/etc/redis.conf <<EOF
bind 0.0.0.0
daemonize yes
pidfile "/usr/local/redis/var/redis-server.pid"
port 6379
tcp-backlog 128
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/usr/local/redis/var/redis-server.log"
databases 16
save 900 1  
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir "/usr/local/redis/data/redis"
masterauth "abc123"                
requirepass "abc123"                     
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly yes                                
appendfilename "appendonly.aof"
appendfsync everysec                         
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
EOF

配置讲解： masterauth 表示master设置密码保护，即slave连接master时的密码 requirepass 表示设置Redis连接密码，如果配置了连接密码，客户端在连接Redis时需要通过AUTH 命令提供密码，默认关闭 appendonly yes 表示打开aof持久化 appendfsync everysec 表示每秒一次aof写

6 从节点redis.conf配置

cat > /usr/local/redis/etc/redis.conf <<EOF
bind 0.0.0.0
daemonize yes
pidfile "/usr/local/redis/var/redis-server.pid"
port 6379
tcp-backlog 128
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/usr/local/redis/var/redis-server.log"
databases 16
save 900 1  
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir "/usr/local/redis/data/redis"
masterauth "abc123"               
requirepass "abc123"      
slaveof k8s-01 6379                        
slave-serve-stale-data yes
slave-read-only yes                          
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly yes                           
appendfilename "appendonly.aof"
appendfsync everysec                        
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
EOF

配置讲解 slaveof 相对主redis配置，多添加了此行
slave-read-only yes 从节点只读，不能写入

7 sentinel.conf配置(所有节点执行)

cat > /usr/local/redis/etc/sentinel.conf<<EOF
port 26379
pidfile "/usr/local/redis/var/redis-sentinel.pid"
dir "/usr/local/redis/data/sentinel"
daemonize yes
protected-mode no
logfile "/usr/local/redis/var/redis-sentinel.log"
sentinel monitor redisMaster k8s-01 6379 2 
sentinel down-after-milliseconds redisMaster 10000 
sentinel parallel-syncs redisMaster 1
sentinel failover-timeout redisMaster 60000 
EOF

8 启动redis和sentinel（所有节点执行）

环境变量设置

cat >> /etc/profile << EOF
export PATH=$PATH:/usr/local/redis/bin
EOF

source /etc/profile

启动redis和sentinel

/etc/init.d/redis-server start
/etc/init.d/redis-sentinel start

验证 ps -ef | grep redis

9 查看redis和sentinel信息

9.1 查看三个节点的redis的主从关系

redis-cli -h k8s-01 -p 6379 -a abc123 INFO |grep role

redis-cli -h k8s-02 -p 6379 -a abc123 INFO |grep role

redis-cli -h k8s-03 -p 6379 -a abc123 INFO |grep role

若k8s-01返回master，另外的返回slave即验证正确

9.2 查看Master节点信息

redis-cli -h k8s-01 -p 6379 -a abc123 INFO Replication

# Replication role:master connected_slaves:2 slave0:ip=10.57.26.8,port=6379,state=online,offset=1862,lag=0 slave1:ip=10.57.26.7,port=6379,state=online,offset=1862,lag=0 master_replid:24655c8434861639e588d91fdfe7044f89328471 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:1862 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:1862

从上面信息看出，此时k8s-01的角色为master，有两个slave（7和8）被连接成功.

此时打开master和slave的sentinel.conf，在末尾可看到如下自动写入的内容：

cat /usr/local/redis/etc/sentinel.conf

# Generated by CONFIG REWRITE sentinel failover-timeout redisMaster 60000 sentinel config-epoch redisMaster 0 sentinel leader-epoch redisMaster 0 sentinel current-epoch 0

redis-cli -h k8s-01 -p 26379 -a abc123 INFO Sentinel

# Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=redisMaster,status=sdown,address=10.57.26.15:6379,slaves=0,sentinels=1

10 写入测试

在master(k8s-01)上执行写入和读取操作，均正常。

在slave(k8s-02)上执行读取操作，正常，执行写入操作，报错readonly，满足预计要求。

由上面测试信息可知，master节点可以写入，可以读取；而slave节点默认只能读取，不能写入！这就实现了主从复制，读写分离了！

11 故障模拟

1. 关掉任意一个slave节点（比如关闭掉slave01节点），所有节点的sentinel都可以检测到，出现如下示例信息： 在k8s-02上，执行shutdown命令关闭redis，可见redis进程已消失。

执行redis-cli -h k8s-01 -p 6379 -a abc123 info replication，如下图所示可看出k8s-02被sentinel检测到已处于关闭状态，此时再来查看剩余节点的主从信息，它们的角色不会发生变化，只是master上的connected_slaves变为了1。

查看sentinel日志（任意节点上查看），发现203节点已经进入”+sdown”状态 tail -f /usr/local/redis/var/redis-sentinel.log

然后重启上面被关闭的slave节点（k8s-02），所有节点的sentinel都可以检测到，可看出又被sentinel检测到已处于可用状态，此时再来查看节点的主从信息， 它们的角色仍然不会发生变化，master上的connected_slaves又变为了2

1. 关掉master节点（即k8s-01），待所有节点的sentinel都检测到后（稍等一会，2-3秒钟时间），再来查看两个Slave节点的主从信息，发现其中一个节点的角色通过选举后会成为 master节点了

这个测试有问题

12 客户端连接redis sentinel

java

pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>redis-test</groupId>
    <artifactId>zhy</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.9.3</version>
        </dependency>
    </dependencies>
</project>

测试文件

package zhy.redis;

import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisSentinelPool;

import java.util.HashSet;
import java.util.Set;

public class RedisTest {
    public static void main(String[] args) {

        Set<String> sentinels = new HashSet<>();
        String hostAndPort1 = "k8s-01:26379";
        sentinels.add(hostAndPort1);

        String clusterName = "redisMaster";
        String password = "abc123";

        JedisSentinelPool redisSentinelJedisPool = new JedisSentinelPool(clusterName, sentinels, password);

        Jedis jedis = null;
        try {
            jedis = redisSentinelJedisPool.getResource();
            jedis.set("key", "value");
            System.out.println(jedis.get("key"));
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            redisSentinelJedisPool.returnBrokenResource(jedis);
        }

        redisSentinelJedisPool.close();
    }
}