05.部署flannel网络

部署环境

IP地址	主机名	功能
10.1.104.200	k8s-deploy	部署节点，不承担实际作用
10.1.104.201	k8s-master01	master节点
10.1.104.202	k8s-master02	master节点
10.1.104.203	k8s-master03	master节点
10.1.104.204	k8s-nginx	负载均衡节点，实际生产中应为HA架构
10.1.104.205	k8s-node01	node节点
10.1.104.206	k8s-node02	node节点
10.1.104.207	k8s-node03	node节点

部署flannel网络

flannel使用vxlan技术为各节点创建一个可以互通的pod网络，使用的端口为UDP的8472。
flannel第一次启动时，从etcd获取配置的pod网段信息，为本节点分配一个未使用的地址段，然后创建flannedl.1网络接口。
flannel将分配给自己的pod网段信息写入/run/flannel/docker文件，docker后续使用这个文件中的环境变量设置docker0网桥，从而从这个地址段为本节点的所有pod容器分配 IP。

获取flannel并分发到各节点（k8s-deploy）：

下载flannel应用程序并解压：

cd /opt/k8s/work
mkdir flannel
wget http://download.wenjun1984.cn/Kubernetes/Flannel/flannel-v0.11.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel

将flannel应用分发到各节点：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
    do
        echo ">>> ${node_ip}"
        scp flannel/{flanneld,mk-docker-opts.sh} root@${node_ip}:/opt/k8s/bin/
        ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
    done

确保每个节点都安装有flannel应用：

for node_ip in ${NODE_IPS[@]}
    do
        echo ">>> ${node_ip}"
        ssh root@${node_ip} "ls -a /opt/k8s/bin/flanneld"
    done

创建flannel证书和私钥（k8s-deploy）：

创建flannel证书签名请求：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

cat > flanneld-csr.json << EOF
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Shanghai",
      "L": "Shanghai",
      "O": "dominos",
      "OU": "ops"
    }
  ]
}
EOF

通过CA生成flannel证书和私钥，其结构如下图：

cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

分发flannel证书和私钥到各节点（k8s-deploy）：

将flannel证书和私钥分发到各节点：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
    do
        echo ">>> ${node_ip}"
        ssh root@${node_ip} "mkdir -p /etc/flanneld/cert"
        scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert
    done

向etcd写入集群pod网段信息（k8s-deploy）：

将pod的网络信息写入etcd中：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/opt/k8s/work/ca.pem \
  --cert-file=/opt/k8s/work/flanneld.pem \
  --key-file=/opt/k8s/work/flanneld-key.pem \
  mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}'

• 写入的pod网段${CLUSTER_CIDR}地址段必须小于SubnetLen，必须与kube-controller-manager的--cluster-cidr参数值一致。

创建flannel的服务文件（k8s-deploy）:

创建flannel配置文件：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
  -etcd-endpoints=${ETCD_ENDPOINTS} \\
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
  -iface=${IFACE} \\
  -ip-masq
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

• flanneld运行时需要root权限。

将flannel的服务文件分发到各节点（k8s-deploy）：

将flannel服务文件分发到各节点：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
    do
        echo ">>> ${node_ip}"
        scp flanneld.service root@${node_ip}:/etc/systemd/system/
    done

启动各节点的flannel服务（k8s-deploy）：

启动各节点的flannel服务：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
    do
        echo ">>> ${node_ip}"
        ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"
    done

确认每个节点的flannel服务是否启动：

for node_ip in ${NODE_IPS[@]}
    do
        echo ">>> ${node_ip}"
        ssh root@${node_ip} "systemctl status flanneld|grep Active"
    done

验证各节点上的flannel状态（k8s-deploy）：

查看创建的pod子网，每个节点的flannel都会创建一个子网：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/opt/k8s/work/ca.pem \
  --cert-file=/opt/k8s/work/flanneld.pem \
  --key-file=/opt/k8s/work/flanneld-key.pem \
  ls ${FLANNEL_ETCD_PREFIX}/subnets

验证各节点能通过pod网络互通（k8s-deploy）：

确认flannel网络接口已经创建：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
    do
        echo ">>> ${node_ip}"
        ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"
    done

查看各节点是否可以ping通flannel接口：

for node_ip in ${NODE_IPS[@]}
    do
        echo ">>> ${node_ip}"
        ssh ${node_ip} "ping -c 1 172.30.232.0"
        ssh ${node_ip} "ping -c 1 172.30.72.0"
        ssh ${node_ip} "ping -c 1 172.30.40.0"
        ssh ${node_ip} "ping -c 1 172.30.80.0"
        ssh ${node_ip} "ping -c 1 172.30.208.0"
        ssh ${node_ip} "ping -c 1 172.30.144.0"
    done