Nginx高可用(利用keepalived非抢占模式来实现)

ginlivenginx

文章目录

第1章整体实践的环境和架构图
- 1.1 我的需求说明
- 1.2 我的实践架构
- 1.3 操作系统环境
- 1.4 实践几个阶段
第2章部署nginx服务且启动它
- 2.1 nginx部署规划说明
- 2.2 node21上部署nginx
- 2.3 node22上部署nginx
第3章部署keepalived并进行测试
- 3.1 keepalived部署规划说明
- 3.2 node21上部署keepalived
- 3.3 node22上部署keepalived
- 3.4 两个keepalived的配置文件对比
- 3.5 只启动node21服务器上的keepalived
- 3.6 启动两台服务器上的keepalived
- 3.7 访问VIP地址看一看结果
- 3.8 停止两台服务器上的keepalived
- 3.9 遗留问题
第4章完善两个keepalived对自身nginx的监控
- 4.1 完善两个keepalived的配置文件
- 4.2 两个keepalvied配置文件的对比
- 4.3 编写检测nginx服务的脚本
- 4.4 启动两个keepalived
- 4.5 进行最终的测试

第1章整体实践的环境和架构图

1.1 我的需求说明

nginx是单点,需要实现高可用,想到的方案就是利用keepalived来实现，当nginx宕服或者nginx所在的服务器宕机（nginx也宕服了）有另一个nginx能够接管起来并向客户端[网站域名的指向、客户端的连接（域名或者IP）]提供服务。在接管时不需要去管备nginx的相关配置，平时在维护nginx的时候，两个nginx的配置都保持一致。

正常情况下(肯定是根据我的规划嘛)

node21服务器上的kee或Ng宕服或 node21服务器宕机

node21服务器上的kee或Ng宕服或 node21服务器宕机进行恢复后

node22服务器上的kee或Ng宕服或 node22服务器宕机

PS：还是得上监控

1
2
3
4
5
6
7

01：在某一台服务器上的kee或者nginx出现故障(宕服/宕机),好在第一时间知道且进行恢复
A：node21服务器上的出故障了,vip会飘移到node22上继续提供服务,你在知道后,第一时间进行恢复,防止node22服务器出故障后业务访问不了
B：node22服务器上的出故障了,此时vip在node21上提供服务,你在知道后,第一时间进行恢复,防止node21服务器出故障后,业务访问不了

02:防止node21或者node22服务器上同时出现故障
A：当同时出现故障,业务访问不了
B:有了监控,你能够在第一时间知道且进行处理。

1.2 我的实践架构

为了更直观的看到当node21服务器宕机、node21服务器上的keepalived宕服、node21服务器上的nginx宕服后，VIP飘移至node22服器上后访问的效果。这里node21服务器和node22服务器上的nginx各自返回的结果是不同的。如下图所示：

正常情况下(肯定是根据我的规划嘛)

node21服务器上的kee或Ng宕服或 node21服务器宕机

node21服务器上的kee或Ng宕服或 node21服务器宕机进行恢复后

node22服务器上的kee或Ng宕服或 node22服务器宕机

1.3 操作系统环境

这里以node21服务器的环境为例,node22服务器的环境也是一样的哈。

1
2
3
4
5
6
7
8
9
10
11
12
13

#### 系统版本
[root@node21 ~]# cat /etc/redhat-release;uname -r;uname -m
CentOS Linux release 7.7.1908 (Core)
3.10.0-1062.el7.x86_64
x86_64

#### firewalld防火墙
[root@node21 ~]# firewall-cmd --state
not running

#### 数据盘
[root@node21 ~]# df -h|grep -w "data"
/dev/sdb 100G 43M 100G 1% /data

1.4 实践几个阶段

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#### 第一阶段: 在node21和node22服务哭上部署且启动nginx
01：网页访问10.0.0.21的结果要是 node21 nginx
02：网页访问10.0.0.22的结果要是 node22 nginx

#### 第二阶段：在node21和node22服务器上部署keepalived
01:在node21和node22服务器上部署keepalived,这两个keepalived都还没nginx做监控
02:只启动node21服务器上的keepalived，进行相关的说明
03:先启动node21服务器上的keepalived,再启动node22服务器上的keepalived,看看效果
04:总结keepalived的非抢占模式

#### 第三阶段：完善node21服务器上的的"主keepalived"对自身nginx的监控
01:停止node21和node22服务上的keepalived
02:完成node21服务器上keepalived对自身的nginx监控;
03:先启动node21服务器上的keepalived和node22服务器上的keepalived;
04:做相关的测试(故意模拟)

第2章部署nginx服务且启动它

2.1 nginx部署规划说明

node21和node22服务器上的nginx部署规划都是一至的哈

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#### nginx程序的安装目录
/data/apps/nginx <== 这是个软链接文件,指定/data/apps/nginx/nginx-1.18.0目录

#### nginx编译安装好以后有如下的目录
[root@node21 ~]# tree -Lp 1 /data/apps/nginx
/data/apps/nginx
├── [drwxr-xr-x] conf <== 里面存放nginx的相关配置文件
├── [drwxr-xr-x] html <== 用来存放相关的静态文件
├── [drwxr-xr-x] logs <== 用来存放日志文件和pid文件
└── [drwxr-xr-x] sbin <== 里面存放的是nginx命令,只有一个命令;

#### 在/data/apps/nginx/目录下创建script目录,用来存放nginx的启停脚本
[root@node21 ~]# tree -Lp 1 /data/apps/nginx
/data/apps/nginx
├── [drwxr-xr-x] conf <== 里面存放nginx的相关配置文件
├── [drwxr-xr-x] html <== 用来存放相关的静态文件
├── [drwxr-xr-x] logs <== 用来存放日志文件和pid文件
└── [drwxr-xr-x] sbin <== 里面存放的是nginx命令,只有一个命令
└── [drwxr-xr-x] script # 这是我人为创建的哈,存放nginx的启停脚本

2.2 node21上部署nginx

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

#### 创建/data/apps目录,用于安装nginx软件
mkdir /data/apps
ls -ld /data/apps

#### 安装相关的依赖包
chattr -i /etc/passwd /etc/group /etc/shadow /etc/gshadow
yum install gcc gcc-c++ openssl-devel pcre-devel -y
rpm -qa gcc gcc-c++ openssl-devel pcre-devel

#### 创建普通用户www,不让其远程登录
chattr -i /etc/passwd /etc/group /etc/shadow /etc/gshadow
useradd -s /sbin/nologin -M www
id www

#### 上传软件,准备编译安装
mkdir /root/tools
cd /root/tools
rz # 若该命令没有,用yum install lrzsz -y进行安装
ll nginx-1.18.0.tar.gz
tar xf nginx-1.18.0.tar.gz
cd nginx-1.18.0

./configure \
--prefix=/data/apps/nginx-1.18.0 \
--user=www \
--group=www \
--with-stream \
--with-http_ssl_module

make && make install

#### 做软链接和配置环境变量
ln -sv /data/apps/nginx-1.18.0/ /data/apps/nginx
chown -R www:www /data/apps/nginx/
echo "PATH=/data/apps/nginx/sbin:$PATH" >>/etc/bashrc
source /etc/bashrc
which nginx

#### 查看nginx的相关目录
[root@node21 ~]# tree -Lp 1 /data/apps/nginx
/data/apps/nginx
├── [drwxr-xr-x] conf
├── [drwxr-xr-x] html
├── [drwxr-xr-x] logs
└── [drwxr-xr-x] sbin

4 directories, 0 files

#### 创建/data/apps/nginx/script/目录，用来存放nginx的启停脚本
mkdir /data/apps/nginx/script
ls -ld /data/apps/nginx/script

#### 对nginx的配置文件做一些简单的处理
[root@node21 ~]# grep -Ev "^$|#" /data/apps/nginx/conf/nginx.conf.default >/data/apps/nginx/conf/nginx.conf
[root@node21 ~]# cat /data/apps/nginx/conf/nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}

#### 对/data/apps/nginx/html/index.html文件做一些处理
echo "node21 nginx" >/data/apps/nginx/html/index.html
cat /data/apps/nginx/html/index.html

#### 编辑nginx的启停脚本
01：vim /data/apps/nginx/script/nginxd文件,按i键进入编辑模式,
02：从 https://blog.csdn.net/weixin_43733154/article/details/107226757 博客复制脚本
内容后,粘贴至/data/apps/nginx/script/nginxd文件中,按esc键回到命令行,输入:wq保存并退出。
03：为/data/apps/nginx/script/nginxd文件加权限
chmod 700 /data/apps/nginx/script/nginxd
ll /data/apps/nginx/script/nginxd

#### 更改属主/组为www用户
chown -R www:www /data/apps/nginx/ # 后面要加左斜杠,因为/data/apps/nginx是软链

#### 启动nginx服务，用root来启动,启动后master线程是root,工作线程是www用户
#### nginx的端口是80,只能用root用户来启动;
/data/apps/nginx/script/nginxd start
netstat -lntup|grep 80

#### 加入开启自启动
echo -ne "\n# Boot start nginx service.USER:chenliang TIME:$(date +%F)\n/data/apps/nginx/script/nginxd start\n" >>/etc/rc.local
tail -2 /etc/rc.local
chmod 700 /etc/rc.d/rc.local

本机访问10.0.0.21 和 172.16.1.21

1
2
3
4

[root@node21 ~]# curl 10.0.0.21
node21 nginx
[root@node21 ~]# curl 172.16.1.21
node21 nginx

网页访问10.0.0.21

2.3 node22上部署nginx

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102

#### 创建/data/apps目录,用于安装nginx软件
mkdir /data/apps
ls -ld /data/apps

#### 安装相关的依赖包
chattr -i /etc/passwd /etc/group /etc/shadow /etc/gshadow
yum install gcc gcc-c++ openssl-devel pcre-devel -y
rpm -qa gcc gcc-c++ openssl-devel pcre-devel

#### 创建普通用户www,不让其远程登录
chattr -i /etc/passwd /etc/group /etc/shadow /etc/gshadow
useradd -s /sbin/nologin -M www
id www

#### 上传软件,准备编译安装
mkdir /root/tools
cd /root/tools
rz # 若该命令没有,用yum install lrzsz -y进行安装
ll nginx-1.18.0.tar.gz
tar xf nginx-1.18.0.tar.gz
cd nginx-1.18.0

./configure \
--prefix=/data/apps/nginx-1.18.0 \
--user=www \
--group=www \
--with-stream \
--with-http_ssl_module

make && make install

#### 做软链接和配置环境变量
ln -sv /data/apps/nginx-1.18.0/ /data/apps/nginx
chown -R www:www /data/apps/nginx/
echo "PATH=/data/apps/nginx/sbin:$PATH" >>/etc/bashrc
source /etc/bashrc
which nginx

#### 查看nginx的相关目录
[root@node22 ~]# tree -Lp 1 /data/apps/nginx
/data/apps/nginx
├── [drwxr-xr-x] conf
├── [drwxr-xr-x] html
├── [drwxr-xr-x] logs
└── [drwxr-xr-x] sbin

4 directories, 0 files

#### 创建/data/apps/nginx/script/目录，用来存放nginx的启停脚本
mkdir /data/apps/nginx/script
ls -ld /data/apps/nginx/script

#### 对nginx的配置文件做一些简单的处理
[root@node22 ~]# grep -Ev "^$|#" /data/apps/nginx/conf/nginx.conf.default >/data/apps/nginx/conf/nginx.conf
[root@node22 ~]# cat /data/apps/nginx/conf/nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}

#### 对/data/apps/nginx/html/index.html文件做一些处理
echo "node22 nginx" >/data/apps/nginx/html/index.html
cat /data/apps/nginx/html/index.html

#### 编辑nginx的启停脚本
01：vim /data/apps/nginx/script/nginxd 文件,按i键进入编辑模式,
02：从 https://blog.csdn.net/weixin_43733154/article/details/107226757 博客复制脚本
内容后,粘贴至/data/apps/nginx/script/nginxd文件中,按esc键回到命令行,输入:wq保存并退出。
03：为/data/apps/nginx/script/nginxd文件加权限
chmod 700 /data/apps/nginx/script/nginxd
ll /data/apps/nginx/script/nginxd

#### 更改属主/组为www用户
chown -R www:www /data/apps/nginx/ # 后面要加左斜杠,因为/data/apps/nginx是软链

#### 启动nginx服务，用root来启动,启动后master线程是root,工作线程是www用户
#### nginx的端口是80,只能用root用户来启动;
/data/apps/nginx/script/nginxd start
netstat -lntup|grep 80

#### 加入开启自启动
echo -ne "\n# Boot start nginx service.USER:chenliang TIME:$(date +%F)\n/data/apps/nginx/script/nginxd start\n" >>/etc/rc.local
tail -2 /etc/rc.local
chmod 700 /etc/rc.d/rc.local

本机访问10.0.0.22 和 172.16.1.22

1
2
3
4

[root@node22 ~]# curl 10.0.0.22
node22 nginx
[root@node22 ~]# curl 172.16.1.22
node22 nginx

网页访问10.0.0.22

第3章部署keepalived并进行测试

3.1 keepalived部署规划说明

node21和node22服务器上对于keepalived的安装规划都是一样的哈

1
2
3
4
5
6
7
8
9
10
11
12
13
14

#### 安装目录
/data/apps/keepalived <== 这是个软链接文件,指定/data/apps/keepalived-2.0.20

#### 安装后的目录和自己创建的目录
[root@node22 ~]# tree -Lp 1 /data/apps/keepalived
/data/apps/keepalived
├── [drwxr-xr-x] bin
├── [drwxr-xr-x] etc
├── [drwxr-xr-x] run <== 该目录用来存放运行的pid文件
├── [drwxr-xr-x] sbin
└── [drwxr-xr-x] share
└── [drwxr-xr-x] script <== 存放检测nginx服务的脚本

6 directories, 0 files

3.2 node21上部署keepalived

这里是把keepalived安装好,配置好(与node21上的keepalived配置不一样哈),但不启动它;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

#### 安装依赖包
chattr -i /etc/passwd /etc/shadow /etc/group /etc/gshadow
yum install -y libnfnetlink-devel zlib zlib-devel gcc gcc-c++ openssl openssl-devel openssh libaio libaio-devel
rpm -qa libnfnetlink-devel zlib zlib-devel gcc gcc-c++ openssl openssl-devel openssh libaio libaio-devel

## 上传keepalived相关软件包并进行安装
ll keepalived-2.0.20.tar.gz
tar xf keepalived-2.0.20.tar.gz
cd keepalived-2.0.20
./configure --prefix=/data/apps/keepalived-2.0.20
make && make install

## 做软链接文件
ln -sv /data/apps/keepalived-2.0.20/ /data/apps/keepalived
ll /data/apps/keepalived

## 创建存放pid的目录
mkdir /data/apps/keepalived/run
ls -ld /data/apps/keepalived/run

## 对默认的配置文件做一个配置
cp -a /data/apps/keepalived/etc/keepalived/keepalived.conf{,.defaults}
ll /data/apps/keepalived/etc/keepalived/keepalived.conf{,.defaults}

## 清空keepalived配置文件的内容,前面做了睾
>/data/apps/keepalived/etc/keepalived/keepalived.conf
cat /data/apps/keepalived/etc/keepalived/keepalived.conf

## 往keepalived(主)配置文件中追加内容, PS:它和node22上的keepalived的配置不一样哈
cat >>/data/apps/keepalived/etc/keepalived/keepalived.conf<<EOF
global_defs {
router_id nginx_lb01
}

vrrp_instance VI_1 {
virtual_router_id 50
state BACKUP
nopreempt
priority 150
advert_int 1
interface ens33
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.20/24 dev ens33 label ens33:20
}
}
EOF

## 编辑keepalived的启停脚本,加入开机自启动,但不启动它
01: vim /data/apps/keepalived/keepd 文件,按i键,进入到编辑模式
02：从 https://blog.csdn.net/weixin_43733154/article/details/109502585 博文件复制
内容粘贴至文件中，按esc键回到命令模式,输入:wq保存并退出
03：修改文件的权限, chmod 700 /data/apps/keepalived/keepd

PS: 这里我没有启动它哈

## 加入开机自启动
echo -ne "# Boost start keepalived(main) service\n/data/apps/keepalived/keepd start\n" >>/etc/rc.local
tail -2 /etc/rc.local
chmod 744 /etc/rc.d/rc.local

3.3 node22上部署keepalived

这里是把keepalived安装好,配置好(与node21上的keepalived配置不一样哈),但不启动它;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

#### 安装依赖包
chattr -i /etc/passwd /etc/shadow /etc/group /etc/gshadow
yum install -y libnfnetlink-devel zlib zlib-devel gcc gcc-c++ openssl openssl-devel openssh libaio libaio-devel
rpm -qa libnfnetlink-devel zlib zlib-devel gcc gcc-c++ openssl openssl-devel openssh libaio libaio-devel

## 上传keepalived相关软件包并进行安装
ll keepalived-2.0.20.tar.gz
tar xf keepalived-2.0.20.tar.gz
cd keepalived-2.0.20
./configure --prefix=/data/apps/keepalived-2.0.20
make && make install

## 做软链接文件
ln -sv /data/apps/keepalived-2.0.20/ /data/apps/keepalived
ll /data/apps/keepalived

## 创建存放pid的目录
mkdir /data/apps/keepalived/run
ls -ld /data/apps/keepalived/run

## 对默认的配置文件做一个配置
cp -a /data/apps/keepalived/etc/keepalived/keepalived.conf{,.defaults}
ll /data/apps/keepalived/etc/keepalived/keepalived.conf{,.defaults}

## 清空keepalived配置文件的内容,前面做了睾
>/data/apps/keepalived/etc/keepalived/keepalived.conf
cat /data/apps/keepalived/etc/keepalived/keepalived.conf

## 往keepalived(主)配置文件中追加内容, PS:它和node21上的keepalived的配置不一样哈
cat >>/data/apps/keepalived/etc/keepalived/keepalived.conf<<EOF
global_defs {
router_id nginx_lb02
}

vrrp_instance VI_1 {
virtual_router_id 50
state BACKUP
nopreempt
priority 100
advert_int 1
interface ens33
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.20/24 dev ens33 label ens33:20
}
}
EOF

## 编辑keepalived的启停脚本,加入开机自启动,但不启动它
01: vim /data/apps/keepalived/keepd 文件,按i键,进入到编辑模式
02：从 https://blog.csdn.net/weixin_43733154/article/details/109502585 博文件复制
内容粘贴至文件中，按esc键回到命令模式,输入:wq保存并退出
03：修改文件的权限, chmod 700 /data/apps/keepalived/keepd

PS: 这里我没有启动它哈

## 加入开机自启动
echo -ne "# Boost start keepalived(prepare) service\n/data/apps/keepalived/keepd start\n" >>/etc/rc.local
tail -2 /etc/rc.local
chmod 744 /etc/rc.d/rc.local

3.4 两个keepalived的配置文件对比

3.5 只启动node21服务器上的keepalived

这里在操作系统的root用户下启动node21服务器上的keepalived，我规划的是"keepalived（节点1）"。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

#### 启动keepalived
[root@node21 ~]# /data/apps/keepalived/keepd start
Start keepalived service [ 确定 ]
[root@node21 ~]# ps -ef|grep keepalived
root 21065 1 0 09:10 ? 00:00:00 /data/apps/keepalived/sbin/keepalived -f /data/apps/keepalived/etc/keepalived/keepalived.conf -p /data/apps/keepalived/run/keepalived.pid
root 21067 21065 0 09:10 ? 00:00:00 /data/apps/keepalived/sbin/keepalived -f /data/apps/keepalived/etc/keepalived/keepalived.conf -p /data/apps/keepalived/run/keepalived.pid
root 21069 1427 0 09:10 pts/0 00:00:00 grep --color=auto keepalived

#### 看一看vip地址是否添加上
[root@node21 ~]# ifconfig ens33:20
ens33:20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.20 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:e5:fc:f9 txqueuelen 1000 (Ethernet)

[root@node21 ~]# ip addr show|grep "10.0.0.20"
inet 10.0.0.20/24 scope global secondary ens33:20

#### 测试vip地址是否正常
[root@node21 ~]# ping -c 2 10.0.0.20|head -3
PING 10.0.0.20 (10.0.0.20) 56(84) bytes of data.
64 bytes from 10.0.0.20: icmp_seq=1 ttl=64 time=0.040 ms
64 bytes from 10.0.0.20: icmp_seq=2 ttl=64 time=0.052 ms

### 说明一：为什么一启动就能够添加上vip地址
01：我的配置文件中进行了配置：10.0.0.20/24 dev ens33 label ens33:20
02：我启动keepalived是用的root用户

### 停止keepalived,看一看vip地址是否会消失
[root@node21 ~]# /data/apps/keepalived/keepd stop
Stop keepalived service [ 确定 ]
[root@node21 ~]#
[root@node21 ~]# ip addr show|grep "10.0.0.20"
[root@node21 ~]#
[root@node21 ~]# ping 10.0.0.20
PING 10.0.0.20 (10.0.0.20) 56(84) bytes of data.
From 10.0.0.21 icmp_seq=1 Destination Host Unreachable

#### 总结:
01：启动keepalived就会添加上vip地址,前提是配置文件中有配置,且启动是用的root用户
02：当keepalived停止时,vip地址会消失(和keepalived的是不是抢占模式和非抢占模式无关);

3.6 启动两台服务器上的keepalived

一定是要先启动node21服务器上的keepalived（节点1），再启动node22服务器上的keepalived（节点2），看1.1和1.2章节，也为了本章该小节后面的相关测试。

启动“keepalived（节点1）”，在node21服务器上操作

1
2
3
4
5
6
7
8
9
10
11

[root@node21 ~]# /data/apps/keepalived/keepd start
Start keepalived service [ 确定 ]
[root@node21 ~]#
[root@node21 ~]# ifconfig ens33:20 # 可以看到是有VIP的哈
ens33:20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.20 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:e5:fc:f9 txqueuelen 1000 (Ethernet)
[root@node21 ~]#
[root@node21 ~]# ping -c 2 10.0.0.20|head -2
PING 10.0.0.20 (10.0.0.20) 56(84) bytes of data.
64 bytes from 10.0.0.20: icmp_seq=1 ttl=64 time=0.019 ms

启动“keepalived（节点2）”，在node22服务器上操作

1
2
3
4
5
6
7
8
9
10
11

[root@node22 ~]# /data/apps/keepalived/keepd start
Start keepalived service [ 确定 ]
[root@node22 ~]#
[root@node22 ~]# ifconfig ens33:20 # 可以看到是没有VIP的哈,正常的
ens33:20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:0c:29:8f:04:d2 txqueuelen 1000 (Ethernet)
[root@node22 ~]#
[root@node22 ~]# ping -c 2 10.0.0.20|head -3 # 当然可ping通,vip在node21服务器上
PING 10.0.0.20 (10.0.0.20) 56(84) bytes of data.
64 bytes from 10.0.0.20: icmp_seq=1 ttl=64 time=0.641 ms
64 bytes from 10.0.0.20: icmp_seq=2 ttl=64 time=0.740 ms

3.7 访问VIP地址看一看结果

此时vip地址在node21服务器上,访问的结果应该是node21 nginx

1
2
3
4
5
6
7
8
9
10
11
12

## 在node21服务器上查看vip地址是否存在(肯定是在的,在上一小节,我是先启动node21服
## 务器上的keepalived)
[root@node21 ~]# ip addr show|grep 10.0.0.20
inet 10.0.0.20/24 scope global secondary ens33:20

## 在node21机器上用curl命令访问10.0.0.20,访问结果是node21 nginx才正确
[root@node21 ~]# curl 10.0.0.20
node21 nginx

## 在node22机器上用curl命令访问10.0.0.20,访问结果是node21 nginx才正确
[root@node22 ~]# curl 10.0.0.20
node21 nginx

现在停止node21服务器上的keepalived(节点1)，vip地址会飘向node22服务器上

1
2
3
4
5
6
7
8
9
10
11
12
13
14

## 在node21服务器上停止keepalived(节点1)
[root@node21 ~]# /data/apps/keepalived/keepd stop
Stop keepalived service [ 确定 ]
[root@node21 ~]#
[root@node21 ~]# ip addr show|grep "10.0.0.20" # 没有vip地址了
[root@node21 ~]#

## 在node22服务器上查看vip地址是否飘移过来
[root@node22 ~]# ip addr show|grep 10.0.0.20 # 已飘移过来
inet 10.0.0.20/24 scope global secondary ens33:20
[root@node22 ~]#
[root@node22 ~]# ping -c 2 10.0.0.20|head -2 # 而且可以ping通
PING 10.0.0.20 (10.0.0.20) 56(84) bytes of data.
64 bytes from 10.0.0.20: icmp_seq=1 ttl=64 time=0.013 ms

此时再来访问10.0.0.20，结果是node22 nginx才对,因为vip已经飘移至node22服务器上

1
2
3
4

[root@node21 ~]# curl 10.0.0.20
node22 nginx
[root@node22 ~]# curl 10.0.0.20
node22 nginx

动node21服务器上的keepalived(节点1),此时看vip能否被抢回来,肯定是不会抢回来的,
因为我们配置的是非抢占模式。没抢回来,你访问10.0.0.20的结果就是node22 nginx了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#### 在node21上启动keepalived（节点1）
[root@node21 ~]# /data/apps/keepalived/keepd start
Start keepalived service [ 确定 ]
[root@node21 ~]#
[root@node21 ~]# ip add show|grep "10.0.0.20" # vip还是没回到node21上
[root@node21 ~]#

### 在node22服务器上看一看vip是否存在,肯定是存在的
[root@node22 ~]# ip addr show|grep 10.0.0.20
inet 10.0.0.20/24 scope global secondary ens33:20

#### 此时你访问10.0.0.20,返回的结果是node22 nginx
[root@node21 ~]# curl 10.0.0.20
node22 nginx
[root@node22 ~]# curl 10.0.0.20
node22 nginx

3.8 停止两台服务器上的keepalived

为了后面"第4章"的相关操作和测试;

1
2
3
4
5
6
7
8
9
10
11

#### node21上的操作
[root@node21 ~]# /data/apps/keepalived/keepd stop
Stop keepalived service [ 确定 ]
[root@node21 ~]# ps -ef|grep "keepalived"|grep -v grep|wc -l
0

#### node22上的操作
[root@node22 ~]# /data/apps/keepalived/keepd stop
Stop keepalived service [ 确定 ]
[root@node22 ~]# ps -ef|grep "keepalived"|grep -v grep|wc -l
0

3.9 遗留问题

目前两个keepalived都没有针对nginx服务的状态做检测，这样就会出现如下的问题：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#### 说明的前提
01："节点1 keepalived" -----非抢占模式-----"节点2 keepalived";
02：这两个节点的keepalived目前都没对自身的nginx服务做检测;
03：假如启动时是先启动的"节点1 keepalived",再启动的"节点2 keepalived";
04：这时vip在"节点1 keepalived"上,客户端访问vip且能够访问到具体的业务;

#### 第一种情况：接"说明的前提"
01：如果"节点1 keepalived"所在服务器上的nginx服务宕服了,那么"节点1 keepalived"是
不知道的,VIP还在"节点1 keepalived"上,这时客户端访问VIP是无法访问到具体业务的;
02：解决方法就是在"节点1 keepalived"所在服务器上编写检测本机nginx服务状态的脚本,
一但检测到nginx未运行,就停止"节点1 keepalived",VIP就会飘移至"节点2
keepalived"上继续向客户端提供服务。

### 第二种情况：接“第一种情况解决了的情况下”
01："节点1 keepalived"宕服或者"节点1 keepalived"所在服务器上的nginx宕服后进行了恢
复,不会把vip从"节点2 keepalived"上抢回来,这是因为keepalived是非抢占模式。
02：如果"节点2 keepalived"所在服务器的nginx宕服了,那么"节点2 keepalived"是不知道
的,vip还在"节点2 keepalived"上,客户端访问vip是无法获取到具体的业务.
03：解决方法就是在"节点2 keepalived"所在服务器上编写检测本机nginx服务的脚本,当
检测到nginx服务未运行时,就停止"节点2 keepalived",此时vip飘移至
"节点1 keepalived"上,继续为客户端提供服务,且客户端能够获取到具体的业务.

### 其实还有问题：不关keepalived的事了
01：如何第一时间知道nginx或keepalived（不管哪台服务器上的）是否宕服;
02：解决方法就是上监控,第一时间知道宕服了要马上进行恢复。本篇文章不涉及

第4章完善两个keepalived对自身nginx的监控

4.1 完善两个keepalived的配置文件

在3.8章节,是把两个keepalived的服务停止了的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

#### node21服务器上keepalived(节点1)配置文件完善
[root@node21 ~]# > /data/apps/keepalived/etc/keepalived/keepalived.conf

cat >>/data/apps/keepalived/etc/keepalived/keepalived.conf<<EOF
global_defs {
router_id nginx_lb01
}

vrrp_script check_ng
{
script "/data/apps/keepalived/script/check_nginx.sh"
interval 2
}

vrrp_instance VI_1 {
virtual_router_id 50
state BACKUP
nopreempt
priority 150
advert_int 1
interface ens33
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.20/24 dev ens33 label ens33:20
}
track_script {
check_ng
}
}
EOF

#### node22服务器上keepalived(节点2)配置文件完善
[root@node22 ~]# > /data/apps/keepalived/etc/keepalived/keepalived.conf

cat >>/data/apps/keepalived/etc/keepalived/keepalived.conf<<EOF
global_defs {
router_id nginx_lb02
}

vrrp_script check_ng
{
script "/data/apps/keepalived/script/check_nginx.sh"
interval 2
}

vrrp_instance VI_1 {
virtual_router_id 50
state BACKUP
nopreempt
priority 100
advert_int 1
interface ens33
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.20/24 dev ens33 label ens33:20
}
track_script {
check_ng
}
}
EOF

4.2 两个keepalvied配置文件的对比

4.3 编写检测nginx服务的脚本

node21上编写检测nginx服务的脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#### 创建存放检测nginx脚本的目录
mkdir /data/apps/keepalived/script
ls -ld /data/apps/keepalived/script

#### 编写脚本
01: vim /data/apps/keepalived/script/check_nginx.sh 后,按i键进入到编辑模式
02：将以下内容复制粘贴至文件中,按esc键后,输入:wq保存并退出

#!/bin/bash
#
# Define variables
RETVAL=0
NginxStats=$(netstat -lntup|grep 80|grep -v grep|wc -l)

## $NginxStats = 0 and stop keepalived
if [ "$NginxStats" -eq 0 ];then
/data/apps/keepalived/keepd stop
fi

#### 添加执行权限
[root@node21 ~]# chmod 700 /data/apps/keepalived/script/check_nginx.sh

node22上编写检测nginx服务的脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#### 创建存放检测nginx脚本的目录
mkdir /data/apps/keepalived/script
ls -ld /data/apps/keepalived/script

#### 编写脚本
01: vim /data/apps/keepalived/script/check_nginx.sh 后,按i键进入到编辑模式
02：将以下内容复制粘贴至文件中,按esc键后,输入:wq保存并退出

#!/bin/bash
#
# Define variables
RETVAL=0
NginxStats=$(netstat -lntup|grep 80|grep -v grep|wc -l)

## $NginxStats = 0 and stop keepalived
if [ "$NginxStats" -eq 0 ];then
/data/apps/keepalived/keepd stop
fi

#### 添加执行权限
[root@node22 ~]# chmod 700 /data/apps/keepalived/script/check_nginx.sh

4.4 启动两个keepalived

先启动node21服务器上的keepalived(节点1),再启动node22服务器上的keepalived(节点
2 )，方便后面的测试。

启动node21服务器上的keepalived(节点1)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

[root@node21 ~]# /data/apps/keepalived/keepd start
Start keepalived service [ 确定 ]
[root@node21 ~]#
[root@node21 ~]# ps -ef|grep "keepalived"|grep -v grep|wc -l
2
[root@node21 ~]#
[root@node21 ~]# tail -2 /var/log/messages # 看日志,是没有报错的
Nov 7 07:30:01 node21 systemd: Started Session 340 of user root.
Nov 7 07:30:08 node21 systemd: Removed slice User Slice of root.
[root@node21 ~]#
[root@node21 ~]# ip addr show|grep 10.0.0.20 # 有vip地址
inet 10.0.0.20/24 scope global secondary ens33:20
[root@node21 ~]#
[root@node21 ~]# ping -c 2 10.0.0.20|head -2
PING 10.0.0.20 (10.0.0.20) 56(84) bytes of data.
64 bytes from 10.0.0.20: icmp_seq=1 ttl=64 time=0.014 ms
[root@node21 ~]#
[root@node21 ~]# curl 10.0.0.20
node21 nginx

启动node22服务器上的keepalived(节点2)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

[root@node22 ~]# /data/apps/keepalived/keepd start
Start keepalived service [ 确定 ]
[root@node22 ~]#
[root@node22 ~]# tail -2 /var/log/messages # 看日志,是没有报错的
Nov 7 07:30:01 node22 systemd: Started Session 338 of user root.
Nov 7 07:30:08 node22 systemd: Removed slice User Slice of root.
[root@node22 ~]#
[root@node22 ~]# ps -ef|grep "keepalived"|grep -v grep|wc -l
2
[root@node22 ~]#
[root@node22 ~]# ip addr show|grep "10.0.0.20" # 没有vip地址
[root@node22 ~]#
[root@node22 ~]# ping -c 2 10.0.0.2|head -2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=128 time=0.145 ms
[root@node22 ~]#
[root@node22 ~]# curl 10.0.0.20
node21 nginx

4.5 进行最终的测试

当前情况,vip在node21服务器上,node21服务器上的keepalived和nginx都运行的,访问vip地址的结果是node21 nginx

1
2
3
4
5
6
7
8
9
10
11

[root@node21 ~]# ps -ef|grep "keepalived"|grep -v grep|wc -l
2
[root@node21 ~]#
[root@node21 ~]# ip addr show|grep "10.0.0.20"
inet 10.0.0.20/24 scope global secondary ens33:20
[root@node21 ~]#
[root@node21 ~]# netstat -lntup|grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 19190/nginx: master
[root@node21 ~]#
[root@node21 ~]# curl 10.0.0.20
node21 nginx

停止node21服务器上的nginx服务,看node21上的keepalived是否会被停止,若能功能停止,vip会飘移至node22服务器上。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#### node21服务器上的操作
[root@node21 ~]# /data/apps/nginx/script/nginxd stop
Stop nginx service [ 确定 ]
[root@node21 ~]#
[root@node21 ~]# ps -ef|grep "keepalived"|grep -v grep|wc -l
0
[root@node21 ~]#
[root@node21 ~]# ip addr show|grep "10.0.0.20" # vip地址没在本机上的
[root@node21 ~]#
[root@node21 ~]# curl 10.0.0.20 # 访问的结果可看出vip已飘移至node22服务器上去了
node22 nginx

#### node22服务器上的查看
[root@node22 ~]# ip addr show|grep 10.0.0.20
inet 10.0.0.20/24 scope global secondary ens33:20
[root@node22 ~]#
[root@node22 ~]# curl 10.0.0.20
node22 nginx

恢复node21服务器上的nginx服务,再恢复node21服务器上的keepalived服务,vip不会被抢加来的,因为配置的是非抢占模式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

[root@node21 ~]# /data/apps/nginx/script/nginxd start
Start nginx service [ 确定 ]
[root@node21 ~]#
[root@node21 ~]# netstat -lntup|grep nginx
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 22002/nginx: master
[root@node21 ~]#
[root@node21 ~]# /data/apps/keepalived/keepd start
Start keepalived service [ 确定 ]
[root@node21 ~]# ps -ef|grep "keepalived"|grep -v "grep"|wc -l
2
[root@node21 ~]#
[root@node21 ~]# ip addr show|grep "10.0.0.20" # 本机没有vip地址
[root@node21 ~]#
[root@node21 ~]# curl 10.0.0.20 # 从结果看vip还是在node22服务器上
node22 nginx

停止node22服务器上的nginx服务, 看node22上的keepalived是否会被停止,若能成功停止,vip会飘移至node22服务器上。

1
2
3
4
5
6
7
8
9
10
11

[root@node22 ~]# /data/apps/nginx/script/nginxd stop
Stop nginx service [ 确定 ]
[root@node22 ~]# netstat -lntup|grep "nginx"|wc -l
0
[root@node22 ~]#
[root@node22 ~]# ps -ef|grep "keepalived"|grep -v grep|wc -l
0
[root@node22 ~]# ip addr show|grep "10.0.0.20" # 本机没有vip地址了
[root@node22 ~]#
[root@node22 ~]# curl 10.0.0.20 # 从结果看vip已经飘移至node21服务器上
node21 nginx

恢复node22服务器上的nginx服务,再恢复keepalived服务,此时vip还是在node21服务器

1
2
3
4
5
6
7
8
9
10
11
12
13
14

[root@node22 ~]# /data/apps/nginx/script/nginxd start
Start nginx service [ 确定 ]
[root@node22 ~]# netstat -lntup|grep nginx
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 28286/nginx: master
[root@node22 ~]#
[root@node22 ~]# /data/apps/keepalived/keepd start
Start keepalived service [ 确定 ]
[root@node22 ~]# ps -ef|grep "keepalived"|grep -v grep|wc -l
2
[root@node22 ~]#
[root@node22 ~]# ip addr show|grep 10.0.0.20
[root@node22 ~]#
[root@node22 ~]# curl 10.0.0.20
node21 nginx