Centos keepalived 双机热备从零开始搭建
阅读 (2416) 2016-12-15 01:45:11
配置不难，过程中会遇到一些问题和注意事项，需要vim、nginx、keepalived等，监测nginx，如果nginx停止工作，则尝试启动nginx，实现高可用
本文为双主轮切，两个真实IP，两个虚拟IP
##虚拟主机A 真实IP
http://10.10.10.175
##虚拟主机B 真实IP
http://10.10.10.176
##虚拟主机A 虚拟IP
http://10.10.10.177
##虚拟主机B 虚拟IP
http://10.10.10.178
实现功能：
访问4个IP中任意IP，均可访问网站，
如果其中一台 keepalived 停止工作，仍可轮切
如果其中一台 nginx 停止工作，执行脚本启动nginx,即使无法启动，两个虚拟IP均访问另一台主机（除停止工作的nginx真实IP无法访问，其它三个IP仍可访问）
开始搭建前需要准备：先实现主机A
1.Linux环境
测试环境:虚拟机 -> CentOS minimal 6.5 64位
IP：用桥接网络，并手动指定IP地址: 10.10.10.175
minimal 版本的CentOS 默认装了精简版的vim 可以直接用vi命令编缉文件
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
如下图所以，BOOTPROTO=dhcp改成status,并添加固定IP网关和子网掩码
配置DNS
vi /etc/resolv.conf
如下图，值可以改成你自己的DNS，当前用的虚拟机测试环境，配置和实体主机相同的DNS即可
其它相关命令：
# 启动网卡
ifup eth0
# 禁用网卡
ifdown eth0
# 重启网络
service network restart
# 开启网络
service network start
# 关闭网络
service network stop
ping 下内网外网是否都正常，这里不细表，具体问题不同解决方法，有问题自行再google
vim安装
网络通了后，安装vim,前面说了默认系统装了精简版的vim，也可以直接用vi命令，自行选择是否要安装vim
yum install vim-enhanced
2.关闭CentOS防火墙
#停止
service iptables stop
#关闭自动启动
chkconfig iptables off
3.彻底关闭SELinux
vim /etc/selinux/config
ESC :wq 保存，退出
reboot 重启系统
4.添加epel源 (Centos官方源很多软件都没有，编译安装又比较麻烦，可以安装EPEL源解决此问题）
最新版本 (可在：http://mirror.centos.org/centos/ 找contos对应版本最新版),
当前测试环境为centos 6.5 64位，所以
rpm -ivh "http://mirror.centos.org/centos/6/extras/x86_64/Packages/epel-release-6-8.noarch.rpm"
5.安装wget
yum -y install wget
6.安装依赖库
yum install bzip2-devel curl-devel db4-devel libjpeg-devel libpng-devel libXpm-devel libc-client-devel libxml2-devel libmcrypt-devel pcre-devel openssl openssl-devel make gcc-c++ cmake bison-devel ncurses-devel gcc gcc-c++ kernel-devel readline-devel pcre-devel openssl-devel openssl zlib zlib-devel pcre-devel libmcrypt libmcrypt-devel
如下图安装完成
开始部署nginx
1.命令安装 nginx下载安装包解压进入目录配置安装
[root@localhost ~]# cd ~
[root@localhost ~]# wget http://nginx.org/download/nginx-1.6.3.tar.gz
[root@localhost ~]# tar zxvf nginx-1.6.3.tar.gz
[root@localhost ~]# cd nginx-1.6.3
[root@localhost ~]# ./configure --prefix=/usr/local/nginx
[root@localhost ~]# make && make install
2.创建nginx启动脚本
vim /etc/init.d/nginx
粘贴以下内容
#!/bin/bash
#
# Startup script for Nginx - this script starts and stops the nginx daemon
#
# chkconfig: - 85 15
# description: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
# processname: nginx
# config: /usr/local/nginx/conf/nginx.conf
# pidfile: /usr/local/nginx/logs/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
nginx="/usr/local/nginx/sbin/nginx"
prog=$(basename $nginx)
NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"
[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
start() {
[ -x $nginx ] || exit 5
[ -f $NGINX_CONF_FILE ] || exit 6
echo -n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
stop() {
echo -n $"Stopping $prog: "
killproc $prog -QUIT
retval=$?
echo
[ $retval -eq 0 ] && rm -f $lockfile
return $retval
}
restart() {
configtest || return $?
stop
sleep 1
start
}
reload() {
configtest || return $?
echo -n $"Reloading $prog: "
killproc $nginx -HUP
RETVAL=$?
echo
}
force_reload() {
restart
}
configtest() {
$nginx -t -c $NGINX_CONF_FILE
}
rh_status() {
status $prog
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case "$1" in
start)
rh_status_q && exit 0
$1
;;
stop)
rh_status_q || exit 0
$1
;;
restart|configtest)
$1
;;
reload)
rh_status_q || exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit 2
esac
ESC :wq保存并退出
3.设置文件权限并设置开机自动启动
[root@localhost nginx-1.6.3]# chmod +x /etc/init.d/nginx
[root@localhost nginx-1.6.3]# chkconfig nginx on
检查一下
[root@localhost nginx-1.6.3]# chkconfig --list nginx
4.修改nginx配置文件
vim /usr/local/nginx/conf/nginx.conf
键入以下内容,内容中涉及另一台主机B，虽然还没有主机B，但可以先写上，后面克隆主机后，只需要修改这里的IP地址，即可实现反转
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
error_log /var/www/logs/error.log info;
#pid logs/nginx.pid;
events {
use epoll;
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
upstream web1 {
server 127.0.0.1:8080 weight=5;
server 10.10.10.176:8080 weight=5; #另外一台主机B真实IP
}
server {
listen 80;
server_name localhost;
root /var/www;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
index index.html index.htm;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_pass http://web1;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
server {
listen 8080;
server_name 10.10.10.175; #当前主机A真实IP
root /var/www;
location / {
index index.html index.htm;
}
}
}
以上配置修改了默认的网站目录
5.创建相关目录（默认网站目录因第4步已修改）
cd /var
mkdir www
cd www
mkdir logs
cp /usr/local/nginx/html/index.html /var/www
6.启动Nginx
service nginx start
相关命令
## 相关命令
service nginx start #开启nginx
service nginx stop #停止nginx
service nginx restart #重启nginx
service nginx reload #重新加载nginx
service nginx status #nginx状态
## 相同效果
/etc/init.d/nginx start
/etc/init.d/nginx stop
/etc/init.d/nginx restart
/etc/init.d/nginx reload
/etc/init.d/nginx status
7.尝试远程访问
做到这里，虚拟机的nginx环境已经搭建好，在真实主机访问：http://10.10.10.175 （改成你自己的测试机IP）, 可以看到下图效果
8.修改nginx默认首页
vim /var/www/index.html
保存并退出，再次回到真实主机在浏览器访问10.10.10.175，可以看到页面也发生了变化
Nginx安装先到这里，后面还有双机热备需要配置另一台主机的Nginx配置，只是反转一下IP
安装Keepalived
1.下载keepalived,解压、配置并安装
[root@localhost ~]# cd ~
[root@localhost ~]# wget http://www.keepalived.org/software/keepalived-1.3.2.tar.gz
[root@localhost ~]# tar -zxvf keepalived-1.3.2.tar.gz
[root@localhost ~]# cd keepalived-1.3.2
[root@localhost keepalived-1.3.2]# ./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-279.el6.x86_64
[root@localhost keepalived-1.3.2]# make && make install
2.创建启动脚本
vim /etc/init.d/keepalived
键入以下内容
#!/bin/sh
#
# Startup script for the Keepalived daemon
#
# processname: keepalived
# pidfile: /var/run/keepalived.pid
# config: /etc/keepalived/keepalived.conf
# chkconfig: - 21 79
# description: Start and stop Keepalived
# Source function library
. /etc/rc.d/init.d/functions
# Source configuration file (we set KEEPALIVED_OPTIONS there)
#. /etc/sysconfig/keepalived
. /usr/local/keepalived/etc/sysconfig/keepalived
PATH="$PATH:/usr/local/keepalived/sbin"
export PAT
RETVAL=0
prog="keepalived"
start() {
echo -n $"Starting $prog: "
daemon keepalived ${KEEPALIVED_OPTIONS}
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
}
stop() {
echo -n $"Stopping $prog: "
killproc keepalived
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog
}
reload() {
echo -n $"Reloading $prog: "
killproc keepalived -1
RETVAL=$?
echo
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
reload)
reload
;;
restart)
stop
start
;;
condrestart)
if [ -f /var/lock/subsys/$prog ]; then
stop
start
fi
;;
status)
status keepalived
RETVAL=$?
;;
*)
echo "Usage: $0 {start|stop|reload|restart|condrestart|status}"
RETVAL=1
esac
exit $RETVAL
权限修改：
chmod +x /etc/init.d/keepalived
3.设置keepalived自启动，并检查
chkconfig keepalived on
chkconfig --list keepalived
如下图显示，自启动正常
4.这里重要了！！！keepalived默认会去找 /etc/keepalived目录下的keepalived.conf配置文件
创建目录，并创建文件
cd /etc
mkdir keepalived
vim /etc/keepalived/keepalived.conf
键入配置内容
! Configuration File for keepalived
global_defs{
notification_email {
email@test.com #填被通知的邮箱地址
}
notification_email_from test@qq.com # 发件人
smtp_server smtp.qq.com
smtp_connect_timeout 30
router_id LVS_212
}
vrrp_script chk_nginx{
script "/usr/local/keepalived/chk_nginx.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
mcast_src_ip 10.10.10.175 #当前主机真实IP
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.10.10.177/24 #当前主机虚拟IP
}
}
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 52
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.10.10.178/24 #另一台主机的虚拟IP
}
}
5.注意到上面的配置中有一段：vrrp_script chk_nginx，里面设置了chk_nginx.sh
我们用这个文件定时检查nginx服务的文件
vim /usr/local/keepalived/chk_nginx.sh
键入以下内容，定时查看nginx是否存在，如果不存在则启动nginx，如果启动失败，则停止keepalived
#!/bin/bash
# description:
# 定时查看nginx是否存在，如果不存在则启动nginx
# 如果启动失败，则停止keepalived
status=$(ps -C nginx --no-heading|wc -l)
if [ "${status}" = "0" ]; then
/usr/local/nginx/sbin/nginx
status2=$(ps -C nginx --no-heading|wc -l)
if [ "${status2}" = "0" ]; then
/etc/init.d/keepalived stop
fi
fi
修改权限：
chmod +x /usr/local/keepalived/chk_nginx.sh
6.相关命令
service keepalived start
service keepalived stop
service keepalived restart
service keepalived reload
/etc/init.d/keepalived start
/etc/init.d/keepalived stop
/etc/init.d/keepalived restart
/etc/init.d/keepalived reload
至此，第一台主机的环境已经搭好，VM虚拟机关机,完整克隆第一台主机,后面修改第二台主机的少许配置即可
另一台主机修改配置
1.修改虚拟机 mac地址和IP地址
vim /etc/sysconfig/network-scripts/ifcfg-eth0
reboot重启
2.修改keepalived.config
! Configuration File for keepalived
global_defs{
notification_email {
zxs@yangxun.com
}
notification_email_from 858785716@qq.com
smtp_server smtp.qq.com
smtp_connect_timeout 30
router_id LVS_212
}
vrrp_script chk_nginx {
script "/usr/local/keepalived/chk_nginx.sh"
interval 2
weight 2
}
## 注：以下内容与主机1不同，注意观察区分
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.10.10.177/24 #另一台主机的虚拟IP
}
}
vrrp_instance VI_2 {
state MASTER
interface eth0
mcast_src_ip 10.10.10.176 #当前主机真实IP
virtual_router_id 52
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
track_script {
chk_nginx
}
virtual_ipaddress {
10.10.10.178/24 #当前主机真实IP
}
}
3.修改nginx配置
vim /usr/local/nginx/conf/nginx.conf
与主机A的配置反转一下
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
error_log /var/www/logs/error.log info;
#pid logs/nginx.pid;
events {
use epoll;
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
upstream web1 {
server 127.0.0.1:8080 weight=5;
server 10.10.10.176:8080 weight=5; #另外一台主机真实IP
}
server {
listen 80;
server_name localhost;
root /var/www;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
index index.html index.htm;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_pass http://web1;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
server {
listen 8080;
server_name 10.10.10.175; #当前主机真实IP
root /var/www;
location / {
index index.html index.htm;
}
}
}
4.修改默认index.html内容，用与区分不同主机
vim /var/www/index.html
把 175 改成 176
到此配置结束试试访问4个地址，查看效果
##虚拟主机A 真实IP
http://10.10.10.175
##虚拟主机B 真实IP
http://10.10.10.176
##虚拟主机A 虚拟IP
http://10.10.10.177
##虚拟主机B 虚拟IP
http://10.10.10.178
用真实主机访问不同IP，再刷新页面，效果应该是 看到175 和176轮换
关闭其中一台主机的nginx，再访问4个地址，
service nginx stop
其中一台主机nginx挂掉
例：停掉主机B的nginx,首先会执行
/usr/local/keepalived/chk_nginx.sh
尝试启动nginx，如果启动失败，则这掉主机B的keepalived
主机A会接管主机B的虚拟IP ，所以仍正常访问
关闭其中一台主机的keepalived，再访问4个地址
service keepalived stop
两台主机中仍有一台主机的keepalived在工作，所以两台主机4个地址仍然工作正常
===========================================
其它问题排查思路
如果keepalived启动出现问题，查看系统日志
cat /var/log/message
如果nginx启动出现问题，检查错误日志
cat /var/www/logs/error.log
更新于：2016-12-15 01:45:11
