07.高可用部署kube-apiserver

部署环境

IP地址	主机名	功能
10.1.32.230	k8s-deploy-test	部署节点，不承担实际作用
10.1.32.231	k8s-master-test01	master节点
10.1.32.232	k8s-master-test02	master节点
10.1.32.233	k8s-master-test03	master节点
10.1.32.240	k8s-nginx-test	负载均衡节点，实际生产中应为HA架构
10.1.32.234	k8s-node01-test01	node节点
10.1.32.235	k8s-node02-test02	node节点
10.1.32.236	k8s-node03-test03	node节点

高可用部署kube-apiserver

Kubernetes API server为api对象验证并配置数据，包括pods、services、replicationcontrollers和其它api对象。
Kubernetes API Server提供REST操作和到集群共享状态的前端，所有其他组件通过它进行交互。

创建kubernetes证书和私钥并分发到各节点（k8s-deploy）：

创建kuber-apiserver证书申请文件：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

cat > kubernetes-csr.json << EOF
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "10.1.32.231",
    "10.1.32.232",
    "10.1.32.233",
    "10.1.32.240",
    "${CLUSTER_KUBERNETES_SVC_IP}",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Shanghai",
      "L": "Shanghai",
      "O": "dominos",
      "OU": "ops"
    }
  ]
}
EOF

hosts字段指定授权使用该证书的IP或域名列表，需要填写vip、每台kube-apiserver节点的、kubernete服务内网域名。
- 域名最后字符不能是.（如不能为 kubernetes.default.svc.cluster.local.），否则解析时失败，会提示：
```
x509: cannot parse dnsName "kubernetes.default.svc.cluster.local."；
```
kubernetes服务ip是apiserver自动创建的，一般是–service-cluster-ip-range参数指定的网段的第一个ip，后续可以通过get svc kubernetes命令获取。

生成kube-apsserver的证书：

cfssl gencert -ca=/opt/k8s/work/ca.pem \
  -ca-key=/opt/k8s/work/ca-key.pem \
  -config=/opt/k8s/work/ca-config.json \
  -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

将kube-apiserver证书分发到各节点：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_MASTER_IPS[@]}
    do
        echo ">>> ${node_ip}"
        ssh root@${node_ip} "mkdir -p /etc/kubernetes/cert"
        scp kubernetes*.pem root@${node_ip}:/etc/kubernetes/cert/
    done

创建加密配置文件并分发到各节点（k8s-deploy）：

创建加密配置文件：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

cat > encryption-config.yaml << EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

将加密配置文件分发到各节点：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_MASTER_IPS[@]}
    do
        echo ">>> ${node_ip}"
        scp encryption-config.yaml root@${node_ip}:/etc/kubernetes/
    done

创建审计策略文件并分发到各节点（k8s-deploy）：

创建审计策略配置文件：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

cat > audit-policy.yaml << EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # The following requests were manually identified as high-volume and low-risk, so drop them.
  - level: None
    resources:
      - group: ""
        resources:
          - endpoints
          - services
          - services/status
    users:
      - 'system:kube-proxy'
    verbs:
      - watch

  - level: None
    resources:
      - group: ""
        resources:
          - nodes
          - nodes/status
    userGroups:
      - 'system:nodes'
    verbs:
      - get

  - level: None
    namespaces:
      - kube-system
    resources:
      - group: ""
        resources:
          - endpoints
    users:
      - 'system:kube-controller-manager'
      - 'system:kube-scheduler'
      - 'system:serviceaccount:kube-system:endpoint-controller'
    verbs:
      - get
      - update

  - level: None
    resources:
      - group: ""
        resources:
          - namespaces
          - namespaces/status
          - namespaces/finalize
    users:
      - 'system:apiserver'
    verbs:
      - get

  # Don't log HPA fetching metrics.
  - level: None
    resources:
      - group: metrics.k8s.io
    users:
      - 'system:kube-controller-manager'
    verbs:
      - get
      - list

  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - '/healthz*'
      - /version
      - '/swagger*'

  # Don't log events requests.
  - level: None
    resources:
      - group: ""
        resources:
          - events

  # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    users:
      - kubelet
      - 'system:node-problem-detector'
      - 'system:serviceaccount:kube-system:node-problem-detector'
    verbs:
      - update
      - patch

  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - nodes/status
          - pods/status
    userGroups:
      - 'system:nodes'
    verbs:
      - update
      - patch

  # deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    omitStages:
      - RequestReceived
    users:
      - 'system:serviceaccount:kube-system:namespace-controller'
    verbs:
      - deletecollection

  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    omitStages:
      - RequestReceived
    resources:
      - group: ""
        resources:
          - secrets
          - configmaps
      - group: authentication.k8s.io
        resources:
          - tokenreviews
  # Get repsonses can be large; skip them.
  - level: Request
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io
    verbs:
      - get
      - list
      - watch

  # Default level for known APIs
  - level: RequestResponse
    omitStages:
      - RequestReceived
    resources:
      - group: ""
      - group: admissionregistration.k8s.io
      - group: apiextensions.k8s.io
      - group: apiregistration.k8s.io
      - group: apps
      - group: authentication.k8s.io
      - group: authorization.k8s.io
      - group: autoscaling
      - group: batch
      - group: certificates.k8s.io
      - group: extensions
      - group: metrics.k8s.io
      - group: networking.k8s.io
      - group: policy
      - group: rbac.authorization.k8s.io
      - group: scheduling.k8s.io
      - group: settings.k8s.io
      - group: storage.k8s.io

  # Default level for all other requests.
  - level: Metadata
    omitStages:
      - RequestReceived
EOF

将审计策略配置文件分发到各节点：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_MASTER_IPS[@]}
    do
        echo ">>> ${node_ip}"
        scp audit-policy.yaml root@${node_ip}:/etc/kubernetes/audit-policy.yaml
    done

创建metrics-sever证书并分发到各节点（k8s-deploy）：

创建metrics-sever证书签名请求：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

cat > proxy-client-csr.json << EOF
{
  "CN": "aggregator",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Shanghai",
      "L": "Shanghai",
      "O": "dominos",
      "OU": "ops"
    }
  ]
}
EOF

CN名称为aggregator，需要与kube-apiserver的启动参数–requestheader-allowed-names参数配置一致，否则访问会被metrics-server拒绝。

创建metrics-server证书文件：

cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client

将metrics-server证书分发到各节点：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_MASTER_IPS[@]}
    do
        echo ">>> ${node_ip}"
        scp proxy-client*.pem root@${node_ip}:/etc/kubernetes/cert/
    done

创建kube-apiserver服务文件并分发到各节点（k8s-deploy）：

创建kube-apiserver服务文件：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

cat > kube-apiserver.service.template << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=${K8S_DIR}/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \\
  --advertise-address=##NODE_IP## \\
  --default-not-ready-toleration-seconds=360 \\
  --default-unreachable-toleration-seconds=360 \\
  --feature-gates=DynamicAuditing=true \\
  --max-mutating-requests-inflight=2000 \\
  --max-requests-inflight=4000 \\
  --default-watch-cache-size=200 \\
  --delete-collection-workers=2 \\
  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\
  --etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
  --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
  --etcd-servers=${ETCD_ENDPOINTS} \\
  --bind-address=##NODE_IP## \\
  --secure-port=6443 \\
  --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
  --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
  --insecure-port=0 \\
  --audit-dynamic-configuration \\
  --audit-log-maxage=15 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-truncate-enabled \\
  --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
  --profiling \\
  --anonymous-auth=false \\
  --client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --enable-bootstrap-token-auth \\
  --enable-aggregator-routing=true \\
  --requestheader-allowed-names="aggregator" \\
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --service-account-key-file=/etc/kubernetes/cert/ca.pem \\
  --authorization-mode=Node,RBAC \\
  --runtime-config=api/all=true \\
  --enable-admission-plugins=NodeRestriction \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --event-ttl=168h \\
  --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\
  --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
  --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
  --kubelet-https=true \\
  --kubelet-timeout=10s \\
  --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\
  --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --service-node-port-range=${NODE_PORT_RANGE} \\
  --logtostderr=true \\
  --enable-aggregator-routing=true \\
  --v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

--advertise-address，apiserver对外通告的ip（kubernetes服务后端节点ip）。
--experimental-encryption-provider-config，指定用于加密etcd中secret的配置。
--etcd-，与etcdu有关的证书和etcd服务器地址。
--bind-address，https监听的ip，不能为127.0.0.1，否则外界不能访问它的安全端口6443。
--secret-port：https监听端口。
--tls-*-file，指定apiserver使用的证书、私钥和ca文件。
--insecure-port=0，关闭监听http非安全端口(8080)。
--audit-*，配置审计策略和审计日志文件相关的参数。
--client-ca-file，验证client(kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书。
--enable-bootstrap-token-auth，启用kubelet bootstrap的token认证。
--requestheader-*，kube-apiserver的aggregator layer相关的配置参数，proxy-client&HPA需要使用。
--requestheader-client-ca-file，用于签名–proxy-client-cert-file和–proxy-client-key-file指定的证书，在启用了metric aggregator时使用。
- 如果--requestheader-allowed-names不为空，则--proxy-client-cert-file证书的CN必须位于allowed-names中，默认为aggregator。
--service-account-key-file，签名ServiceAccount Token的公钥文件。
- kube-controller-manager的--service-account-private-key-file，指定私钥文件，两者配对使用；
--runtime-config=api/all=true，启用所有版本的 APIs，如 autoscaling/v2alpha1。
--authorization-mode=Node,RBAC，开启Node和RBAC授权模式。
--anonymous-auth=false，拒绝未授权的请求。
--enable-admission-plugins，启用一些默认关闭的plugins。
--allow-privileged，运行执行 privileged 权限的容器。
--apiserver-count=3，指定 apiserver 实例的数量。
--event-ttl，指定events的保存时间。
--kubelet-*，如果指定，则使用https访问kubelet APIs；需要为证书对应的用户(上面 kubernetes*.pem证书的用户为kubernetes)用户定义RBAC规则，否则访问 kubelet API时提示未授权。
--proxy-client-*，apiserver访问metrics-server使用的证书。
--service-cluster-ip-range，指定Service Cluster IP地址段。
--service-node-port-range，指定NodePort的端口范围。
--enable-aggregator-routing=true，如果kube-apiserver机器没有运行kube-proxy，则需要添加该参数。
--enable-aggregator-routing=true，Aggregation允许在不修改Kubernetes核心代码的同时扩展Kubernetes API；1.17版本以后需要添加此参数。

生成每个节点的kube-apiserver的服务文件：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for (( i=0; i < 3; i++ ))
    do
        sed -e "s/##NODE_NAME##/${NODE_MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${NODE_MASTER_IPS[i]}.service 
    done

将kube-apiserver的服务文件分发到各节点：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_MASTER_IPS[@]}
    do
        echo ">>> ${node_ip}"
        scp kube-apiserver-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-apiserver.service
    done

启动各节点的kube-apiserver服务（k8s-deploy）：

启动各节点的kube-apiserver服务：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_MASTER_IPS[@]}
    do
        echo ">>> ${node_ip}"
        ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"
        ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
    done

检测各节点的kube-apiserver服务是否启动：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_MASTER_IPS[@]}
    do
        echo ">>> ${node_ip}"
        ssh root@${node_ip} "systemctl status kube-apiserver |grep 'Active:'"
    done

从ectd中获取kube-apiserver组件信息：

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh

ETCDCTL_API=3 etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--cacert=/opt/k8s/work/ca.pem \
--cert=/opt/k8s/work/etcd.pem \
--key=/opt/k8s/work/etcd-key.pem \
get /registry/ --prefix --keys-only

# 在任意一台kube-apiserver上执行，查看组件信息
kubectl get componentstatuses

1.16版本的apiserver有bug，导致kubectl get cs显示为unknow，使用-o yaml则可以获取正常的输出；1.17版本正常。

授予各组件访问kube-apiserver的权限（k8s-master）：

授予kubectl下的命令访问kube-apiserver的权限：

kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

文档更新时间: 2021-09-02 17:22 作者：闻骏