06.部署nginx反向代理

部署环境

IP地址	主机名	功能
10.1.104.200	k8s-deploy	部署节点，不承担实际作用
10.1.104.201	k8s-master01	master节点
10.1.104.202	k8s-master02	master节点
10.1.104.203	k8s-master03	master节点
10.1.104.204	k8s-nginx	负载均衡节点，实际生产中应为HA架构
10.1.104.205	k8s-node01	node节点
10.1.104.206	k8s-node02	node节点
10.1.104.207	k8s-node03	node节点

部署nginx反向代理

nginx反向代理近用于kube-apiserver的负载均衡，kubectl等应用调用kube-apiserver时都会使用负载均衡地址。

部署nginx（k8s-nginx，这里nginx应该是高可用架构，这里仅做单机演示）：

创建nginx用户及用户组：

groupadd nginx && useradd -r -g nginx nginx

安装各依赖包：

yum -y install gcc gcc-c++ glibc
yum -y install gcc gcc-c++ glibc openssl openssl-devel pcre-devel zlib-devel
yum -y group install "Development Tools"

获取及编译安装Nginx：

cd /opt/Downloads
wget http://download.wenjun1984.cn/Nginx/nginx-1.16.0.tar.gz
tar -zxvf nginx-1.16.0.tar.gz && cd nginx-1.16.0

cd /opt/Downloads/nginx-1.16.0
./configure --user=nginx --group=nginx \
--prefix=/opt/Apps/nginx \
--pid-path=/opt/Apps/nginx/nginx.pid \
--lock-path=/opt/Apps/nginx/nginx.lock \
--with-http_ssl_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_sub_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_mp4_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_random_index_module \
--with-http_secure_link_module \
--with-http_stub_status_module \
--with-http_auth_request_module \
--with-threads \
--with-stream \
--with-stream_ssl_module \
--with-http_slice_module \
--with-mail \
--with-mail_ssl_module \
--with-file-aio \
--with-http_v2_module

make && make install

• --with-stream，stream模块，用于开启四层转发功能，此模块必须安装。

配置nginx（k8s-nginx）：

创建配置文件子目录并编写Nginx配置文件：

mkdir /opt/Apps/nginx/conf/conf.d
cat > /opt/Apps/nginx/conf/nginx.conf << "EOF"
user nginx nginx;
worker_processes 2;
worker_rlimit_nofile 65536;

events {
    use epoll;
    worker_connections 10240;
    }

http {
    include mime.types;
    default_type application/octet-stream;
    sendfile on;
    server_names_hash_bucket_size 128;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 32k;
    client_max_body_size 8m;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;

    log_format main  
        '{"@timestamp":"$time_iso8601",'
        '"@source":"$server_addr",'
        '"hostname":"$hostname",'
        '"ip":"$http_x_forwarded_for",'
        '"client":"$remote_addr",'
        '"request_method":"$request_method",'
        '"scheme":"$scheme",'
        '"domain":"$server_name",'
        '"referer":"$http_referer",'
        '"request":"$request_uri",'
        '"args":"$args",'
        '"size":$body_bytes_sent,'
        '"status": $status,'
        '"responsetime":$request_time,'
        '"upstreamtime":"$upstream_response_time",'
        '"upstreamaddr":"$upstream_addr",'
        '"http_user_agent":"$http_user_agent",'
        '"https":"$https"'
        '}';

    fastcgi_connect_timeout 300;
    fastcgi_send_timeout 300;
    fastcgi_read_timeout 300;
    fastcgi_buffer_size 64k;
    fastcgi_buffers 4 64k;
    fastcgi_busy_buffers_size 128k;
    fastcgi_temp_file_write_size 128k;
    fastcgi_cache_path /opt/Apps/nginx/nginx_fastcgi levels=1:2 keys_zone=fastcgi:30m max_size=100m;

    proxy_connect_timeout 120;
    proxy_send_timeout 120;
    proxy_read_timeout 120;
    proxy_buffer_size 64k;
    proxy_buffers 4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_cache_path /opt/Apps/nginx/nginx_proxy levels=1:2 keys_zone=proxy:30m max_size=100m;

    gzip on;
    gzip_min_length 1k;
    gzip_buffers 4 16k;
    gzip_http_version 1.1;
    gzip_comp_level 9;
    gzip_types text/plain application/x-javascript text/css application/xml image/png image/jpeg image/gif audio/ogg audio/mp3;
    gzip_vary on;

    include /opt/Apps/nginx/conf/conf.d/*.conf;
    }

stream {
    log_format proxy '$remote_addr [$time_local]'
        '$protocol $status $bytes_sent $bytes_received'
        '$session_time "$upstream_addr"'
        '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

    include /opt/Apps/nginx/conf/conf.d/*.stream;
    }
EOF

• ”EOF”，EOF两边加引号或之前加”\“转义符，可避免对文本内的特殊符号进行替换（如变量）。

创建kube-apiserver的四层负载均衡配置文件：

cat > /opt/Apps/nginx/conf/conf.d/api-server.stream << "EOF"
upstream backend {
    hash $remote_addr consistent;
    server 10.1.104.201:6443 max_fails=3 fail_timeout=30s;
    server 10.1.104.202:6443 max_fails=3 fail_timeout=30s;
    server 10.1.104.203:6443 max_fails=3 fail_timeout=30s;
}

server {
    listen 8443;

    error_log logs/apiserver_error.log error;
    access_log logs/apiserver_access.log proxy;
    proxy_connect_timeout 3s;
    proxy_timeout 3600s;
    proxy_pass backend;
}
EOF

• upstream backend中的server为真实的kube-apiserver的地址加端口。

编写Nginx的服务文件：

cat > /usr/lib/systemd/system/nginx.service << EOF
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/opt/Apps/nginx/nginx.pid
ExecStartPre=/usr/bin/rm -f /opt/Apps/nginx/nginx.pid
ExecStartPre=/opt/Apps/nginx/sbin/nginx -t
ExecStart=/opt/Apps/nginx/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=process
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

设置Nginx开机自启并启动Nginx：

systemctl enable nginx.service
systemctl start nginx.service
netstat -ntlp | grep 8443