部署环境
| IP地址 |
主机名 |
功能 |
|---|---|---|
| 10.1.104.200 | k8s-deploy | 部署节点，不承担实际作用 |
| 10.1.104.201 | k8s-master01 | master节点 |
| 10.1.104.202 | k8s-master02 | master节点 |
| 10.1.104.203 | k8s-master03 | master节点 |
| 10.1.104.204 | k8s-nginx | 负载均衡节点，实际生产中应为HA架构 |
| 10.1.104.205 | k8s-node01 | node节点 |
| 10.1.104.206 | k8s-node02 | node节点 |
| 10.1.104.207 | k8s-node03 | node节点 |
高可用部署kube-apiserver
Kubernetes API server为api对象验证并配置数据，包括pods、services、replicationcontrollers和其它api对象。
Kubernetes API Server提供REST操作和到集群共享状态的前端，所有其他组件通过它进行交互。
创建kubernetes证书和私钥并分发到各节点（k8s-deploy）：
创建kuber-apiserver证书申请文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kubernetes-csr.json << EOF
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"10.1.104.201",
"10.1.104.202",
"10.1.104.203",
"10.1.104.204",
"${CLUSTER_KUBERNETES_SVC_IP}",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Shanghai",
"L": "Shanghai",
"O": "dominos",
"OU": "ops"
}
]
}
EOF- hosts字段指定授权使用该证书的IP或域名列表，需要填写vip、每台kube-apiserver节点的、kubernete服务内网域名。
- 域名最后字符不能是.（如不能为 kubernetes.default.svc.cluster.local.），否则解析时失败，会提示：
x509: cannot parse dnsName "kubernetes.default.svc.cluster.local."；
- 域名最后字符不能是.（如不能为 kubernetes.default.svc.cluster.local.），否则解析时失败，会提示：
- kubernetes服务ip是apiserver自动创建的，一般是–service-cluster-ip-range参数指定的网段的第一个ip，后续可以通过get svc kubernetes命令获取。
生成kube-apsserver的证书：
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
将kube-apiserver证书分发到各节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p /etc/kubernetes/cert"
scp kubernetes*.pem root@${node_ip}:/etc/kubernetes/cert/
done创建加密配置文件并分发到各节点（k8s-deploy）：
创建加密配置文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > encryption-config.yaml << EOF
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: ${ENCRYPTION_KEY}
- identity: {}
EOF将加密配置文件分发到各节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
scp encryption-config.yaml root@${node_ip}:/etc/kubernetes/
done创建审计策略文件并分发到各节点（k8s-deploy）：
创建审计策略配置文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > audit-policy.yaml << EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
# The following requests were manually identified as high-volume and low-risk, so drop them.
- level: None
resources:
- group: ""
resources:
- endpoints
- services
- services/status
users:
- 'system:kube-proxy'
verbs:
- watch
- level: None
resources:
- group: ""
resources:
- nodes
- nodes/status
userGroups:
- 'system:nodes'
verbs:
- get
- level: None
namespaces:
- kube-system
resources:
- group: ""
resources:
- endpoints
users:
- 'system:kube-controller-manager'
- 'system:kube-scheduler'
- 'system:serviceaccount:kube-system:endpoint-controller'
verbs:
- get
- update
- level: None
resources:
- group: ""
resources:
- namespaces
- namespaces/status
- namespaces/finalize
users:
- 'system:apiserver'
verbs:
- get
# Don't log HPA fetching metrics.
- level: None
resources:
- group: metrics.k8s.io
users:
- 'system:kube-controller-manager'
verbs:
- get
- list
# Don't log these read-only URLs.
- level: None
nonResourceURLs:
- '/healthz*'
- /version
- '/swagger*'
# Don't log events requests.
- level: None
resources:
- group: ""
resources:
- events
# node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- nodes/status
- pods/status
users:
- kubelet
- 'system:node-problem-detector'
- 'system:serviceaccount:kube-system:node-problem-detector'
verbs:
- update
- patch
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- nodes/status
- pods/status
userGroups:
- 'system:nodes'
verbs:
- update
- patch
# deletecollection calls can be large, don't log responses for expected namespace deletions
- level: Request
omitStages:
- RequestReceived
users:
- 'system:serviceaccount:kube-system:namespace-controller'
verbs:
- deletecollection
# Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
# so only log at the Metadata level.
- level: Metadata
omitStages:
- RequestReceived
resources:
- group: ""
resources:
- secrets
- configmaps
- group: authentication.k8s.io
resources:
- tokenreviews
# Get repsonses can be large; skip them.
- level: Request
omitStages:
- RequestReceived
resources:
- group: ""
- group: admissionregistration.k8s.io
- group: apiextensions.k8s.io
- group: apiregistration.k8s.io
- group: apps
- group: authentication.k8s.io
- group: authorization.k8s.io
- group: autoscaling
- group: batch
- group: certificates.k8s.io
- group: extensions
- group: metrics.k8s.io
- group: networking.k8s.io
- group: policy
- group: rbac.authorization.k8s.io
- group: scheduling.k8s.io
- group: settings.k8s.io
- group: storage.k8s.io
verbs:
- get
- list
- watch
# Default level for known APIs
- level: RequestResponse
omitStages:
- RequestReceived
resources:
- group: ""
- group: admissionregistration.k8s.io
- group: apiextensions.k8s.io
- group: apiregistration.k8s.io
- group: apps
- group: authentication.k8s.io
- group: authorization.k8s.io
- group: autoscaling
- group: batch
- group: certificates.k8s.io
- group: extensions
- group: metrics.k8s.io
- group: networking.k8s.io
- group: policy
- group: rbac.authorization.k8s.io
- group: scheduling.k8s.io
- group: settings.k8s.io
- group: storage.k8s.io
# Default level for all other requests.
- level: Metadata
omitStages:
- RequestReceived
EOF将审计策略配置文件分发到各节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
scp audit-policy.yaml root@${node_ip}:/etc/kubernetes/audit-policy.yaml
done创建metrics-sever证书并分发到各节点（k8s-deploy）：
创建metrics-sever证书签名请求：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > proxy-client-csr.json << EOF
{
"CN": "aggregator",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Shanghai",
"L": "Shanghai",
"O": "dominos",
"OU": "ops"
}
]
}
EOF- CN名称为aggregator，需要与kube-apiserver的启动参数–requestheader-allowed-names参数配置一致，否则访问会被metrics-server拒绝。
创建metrics-server证书文件：
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-client
将metrics-server证书分发到各节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
scp proxy-client*.pem root@${node_ip}:/etc/kubernetes/cert/
done创建kube-apiserver服务文件并分发到各节点（k8s-deploy）：
创建kube-apiserver服务文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-apiserver.service.template << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=${K8S_DIR}/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \\
--advertise-address=##NODE_IP## \\
--default-not-ready-toleration-seconds=360 \\
--default-unreachable-toleration-seconds=360 \\
--feature-gates=DynamicAuditing=true \\
--max-mutating-requests-inflight=2000 \\
--max-requests-inflight=4000 \\
--default-watch-cache-size=200 \\
--delete-collection-workers=2 \\
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\
--etcd-cafile=/etc/kubernetes/cert/ca.pem \\
--etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
--etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
--etcd-servers=${ETCD_ENDPOINTS} \\
--bind-address=##NODE_IP## \\
--secure-port=6443 \\
--tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
--tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
--insecure-port=0 \\
--audit-dynamic-configuration \\
--audit-log-maxage=15 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-truncate-enabled \\
--audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \\
--profiling \\
--anonymous-auth=false \\
--client-ca-file=/etc/kubernetes/cert/ca.pem \\
--enable-bootstrap-token-auth \\
--enable-aggregator-routing=true \\
--requestheader-allowed-names="aggregator" \\
--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
--requestheader-extra-headers-prefix="X-Remote-Extra-" \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--service-account-key-file=/etc/kubernetes/cert/ca.pem \\
--authorization-mode=Node,RBAC \\
--runtime-config=api/all=true \\
--enable-admission-plugins=NodeRestriction \\
--allow-privileged=true \\
--apiserver-count=3 \\
--event-ttl=168h \\
--kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\
--kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
--kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
--kubelet-https=true \\
--kubelet-timeout=10s \\
--proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\
--proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\
--service-cluster-ip-range=${SERVICE_CIDR} \\
--service-node-port-range=${NODE_PORT_RANGE} \\
--logtostderr=true \\
--v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF- --advertise-address，apiserver对外通告的ip（kubernetes服务后端节点ip）。
- --experimental-encryption-provider-config，指定用于加密etcd中secret的配置。
- --etcd-，与etcdu有关的证书和etcd服务器地址。
- --bind-address，https监听的ip，不能为127.0.0.1，否则外界不能访问它的安全端口6443。
- --secret-port：https监听端口。
- --tls-*-file，指定apiserver使用的证书、私钥和ca文件。
- --insecure-port=0，关闭监听http非安全端口(8080)。
- --audit-*，配置审计策略和审计日志文件相关的参数。
- --client-ca-file，验证client(kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书。
- --enable-bootstrap-token-auth，启用kubelet bootstrap的token认证。
- --requestheader-*，kube-apiserver的aggregator layer相关的配置参数，proxy-client&HPA需要使用。
- --requestheader-client-ca-file，用于签名–proxy-client-cert-file和–proxy-client-key-file指定的证书，在启用了metric aggregator时使用。
- 如果--requestheader-allowed-names不为空，则--proxy-client-cert-file证书的CN必须位于allowed-names中，默认为aggregator。
- --service-account-key-file，签名ServiceAccount Token的公钥文件。
- kube-controller-manager的--service-account-private-key-file，指定私钥文件，两者配对使用；
- --runtime-config=api/all=true，启用所有版本的 APIs，如 autoscaling/v2alpha1。
- --authorization-mode=Node,RBAC，开启Node和RBAC授权模式。
- --anonymous-auth=false，拒绝未授权的请求。
- --enable-admission-plugins，启用一些默认关闭的plugins。
- --allow-privileged，运行执行 privileged 权限的容器。
- --apiserver-count=3，指定 apiserver 实例的数量。
- --event-ttl，指定events的保存时间。
- --kubelet-*，如果指定，则使用https访问kubelet APIs；需要为证书对应的用户(上面 kubernetes*.pem证书的用户为kubernetes)用户定义RBAC规则，否则访问 kubelet API时提示未授权。
- --proxy-client-*，apiserver访问metrics-server使用的证书。
- --service-cluster-ip-range，指定Service Cluster IP地址段。
- --service-node-port-range，指定NodePort的端口范围。
- --enable-aggregator-routing=true，如果kube-apiserver机器没有运行kube-proxy，则需要添加该参数。
生成每个节点的kube-apiserver的服务文件：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
do
sed -e "s/##NODE_NAME##/${NODE_MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${NODE_MASTER_IPS[i]}.service
done
将kube-apiserver的服务文件分发到各节点：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
scp kube-apiserver-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-apiserver.service
done启动各节点的kube-apiserver服务（k8s-deploy）：
启动各节点的kube-apiserver服务：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"
ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"
done检测各节点的kube-apiserver服务是否启动：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_MASTER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "systemctl status kube-apiserver |grep 'Active:'"
done从ectd中获取kube-apiserver组件信息：
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
ETCDCTL_API=3 etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--cacert=/opt/k8s/work/ca.pem \
--cert=/opt/k8s/work/etcd.pem \
--key=/opt/k8s/work/etcd-key.pem \
get /registry/ --prefix --keys-only
# 在任意一台kube-apiserver上执行，查看组件信息
kubectl get componentstatuses授予各组件访问kube-apiserver的权限（k8s-master）：
授予kubectl下的命令访问kube-apiserver的权限：
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes文档更新时间: 2020-10-22 18:54 作者：闻骏
