个人名片：
因为云计算成为了监控工程师
个人博客：念舒_C.ying
CSDN主页️：念舒_C.ying
部署kube-apiserver集群
10.1 创建kube-apiserver 证书
10.2 创建加密配置文件
10.3 创建 Kubernetes webhook 证书
10.4 创建 kube-apiserver 配置文件
10.5 分发kube-apiserver 证书及配置
10.6 创建 kube-apiserver systemd unit 文件
10.7 启动 kube-apiserver 服务
10.8 检查启动结果
10.9 验证服务状态
集群规划：
服务网段：10.66.0.0/16
Pod 网段：10.80.0.0/12
集群域名：cluster.local
10.1 创建kube-apiserver 证书
创建证书签名请求：
cd /opt/k8s/work
cat > /opt/k8s/cfssl/k8s/k8s-apiserver.json << EOF
{
"CN": "kubernetes",
"hosts": [
"192.168.2.175","192.168.2.176","192.168.2.177",
"10.66.0.1",
"192.168.2.175","127.0.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "GuangZhou",
"O": "k8s",
"OU": "Qist"
}
]
}
EOF
hosts 字段指定授权使用该证书的 IP 和域名列表，这里列出了 master 节点 IP、kubernetes 服务的IP 和域名；
10.66.0.1：kube-apiserver service ip 一般是service第一个ip service-cluster-ip-range 参数
“192.168.2.175”,“192.168.2.176”,“192.168.2.177”： master 节点IP
“192.168.2.175”,“127.0.0.1”：192.168.2.175 vip ip 方便客户端访问 本地127IP 能访问
kube-ha-proxy使用"kubernetes.default.svc.cluster.local"：全局域名访问cluster.local 可以是其它域
生成 Kubernetes API Server 证书和私钥
cfssl gencert \
-ca=/opt/k8s/cfssl/pki/k8s/k8s-ca.pem \
-ca-key=/opt/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
-config=/opt/k8s/cfssl/ca-config.json \
-profile=kubernetes \
/opt/k8s/cfssl/k8s/k8s-apiserver.json | \
cfssljson -bare /opt/k8s/cfssl/pki/k8s/k8s-server
10.2 创建加密配置文件
# 生成 EncryptionConfig 所需的加密 key
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
cd /opt/k8s/work
mkdir config
cat > config/encryption-config.yaml << EOF
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: ${ENCRYPTION_KEY}
- identity: {}
EOF
10.3 创建 Kubernetes webhook 证书
创建证书签名请求：
cd /opt/k8s/work
cat > /opt/k8s/cfssl/k8s/aggregator.json << EOF
{
"CN": "aggregator",
"hosts": [""],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "GuangZhou",
"O": "k8s",
"OU": "Qist"
}
]
}
EOF
CN 名称需要位于 kube-apiserver 的 --requestheader-allowed-names
参数中，否则后续访问metrics 时会提示权限不足。
生成 Kubernetes webhook 证书和私钥
cfssl gencert \
-ca=/opt/k8s/cfssl/pki/k8s/k8s-ca.pem \
-ca-key=/opt/k8s/cfssl/pki/k8s/k8s-ca-key.pem \
-config=/opt/k8s/cfssl/ca-config.json \
-profile=kubernetes \
/opt/k8s/cfssl/k8s/aggregator.json | \
cfssljson -bare /opt/k8s/cfssl/pki/k8s/aggregator
10.4 创建 kube-apiserver 配置文件
192.168.2.175节点：
k8s-master-1 节点上执行
cat >/apps/k8s/conf/kube-apiserver <<EOF
KUBE_APISERVER_OPTS="--logtostderr=true \
--bind-address=192.168.2.175 \
--advertise-address=192.168.2.175 \
--secure-port=5443 \
--insecure-port=0 \
--service-cluster-ip-range=10.66.0.0/16 \
--service-node-port-range=30000-65535 \
--etcd-cafile=/apps/k8s/ssl/etcd/etcd-ca.pem \
--etcd-certfile=/apps/k8s/ssl/etcd/etcd-client.pem \
--etcd-keyfile=/apps/k8s/ssl/etcd/etcd-client-key.pem \
--etcd-prefix=/registry \
--etcdservers=https://192.168.2.175:2379,https://192.168.2.176:2379,https://192.168.2.177:2
379 \
--client-ca-file=/apps/k8s/ssl/k8s/k8s-ca.pem \
--tls-cert-file=/apps/k8s/ssl/k8s/k8s-server.pem \
--tls-private-key-file=/apps/k8s/ssl/k8s/k8s-server-key.pem \
--kubelet-client-certificate=/apps/k8s/ssl/k8s/k8s-server.pem \
--kubelet-client-key=/apps/k8s/ssl/k8s/k8s-server-key.pem \
--service-account-key-file=/apps/k8s/ssl/k8s/k8s-ca.pem \
--requestheader-client-ca-file=/apps/k8s/ssl/k8s/k8s-ca.pem \
--proxy-client-cert-file=/apps/k8s/ssl/k8s/aggregator.pem \
--proxy-client-key-file=/apps/k8s/ssl/k8s/aggregator-key.pem \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--service-account-signing-key-file=/apps/k8s/ssl/k8s/k8s-ca-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true \
--anonymous-auth=false \
--experimental-encryption-provider-config=/apps/k8s/config/encryptionconfig.yaml \
--enable-admissionplugins=DefaultStorageClass,DefaultTolerationSeconds,LimitRanger,NamespaceExists,Name
spaceLifecycle,NodeRestriction,PodNodeSelector,PersistentVolumeClaimResize,PodTolerat
ionRestriction,ResourceQuota,ServiceAccount,StorageObjectInUseProtection,MutatingAdmi
ssionWebhook,ValidatingAdmissionWebhook \
--disable-admissionplugins=ExtendedResourceToleration,ImagePolicyWebhook,LimitPodHardAntiAffinityTopolog
y,NamespaceAutoProvision,Priority,EventRateLimit,PodSecurityPolicy \
--cors-allowed-origins=.* \
--enable-swagger-ui \
--runtime-config=api/all=true \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--authorization-mode=Node,RBAC \
--allow-privileged=true \
--apiserver-count=3 \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--default-not-ready-toleration-seconds=30 \
--default-unreachable-toleration-seconds=30 \
--audit-log-truncate-enabled \
--audit-log-path=/apps/k8s/log/api-server-audit.log \
--profiling \
--http2-max-streams-per-connection=10000 \
--event-ttl=1h \
--enable-bootstrap-token-auth=true \
--alsologtostderr=true \
--log-dir=/apps/k8s/log \
--v=2 \
--tls-ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDH
E_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES
_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 \
--endpoint-reconciler-type=lease \
--max-mutating-requests-inflight=500 \
--max-requests-inflight=1500 \
--target-ram-mb=300"
EOF
192.168.2.176节点：
k8s-master-2 节点上执行
cat >/apps/k8s/conf/kube-apiserver <<EOF
KUBE_APISERVER_OPTS="--logtostderr=true \
--bind-address=192.168.2.176 \
--advertise-address=192.168.2.176 \
--secure-port=5443 \
--insecure-port=0 \
--service-cluster-ip-range=10.66.0.0/16 \
--service-node-port-range=30000-65535 \
--etcd-cafile=/apps/k8s/ssl/etcd/etcd-ca.pem \
--etcd-certfile=/apps/k8s/ssl/etcd/etcd-client.pem \
--etcd-keyfile=/apps/k8s/ssl/etcd/etcd-client-key.pem \
--etcd-prefix=/registry \
--etcdservers=https://192.168.2.175:2379,https://192.168.2.176:2379,https://192.168.2.177:2
379 \
--client-ca-file=/apps/k8s/ssl/k8s/k8s-ca.pem \
--tls-cert-file=/apps/k8s/ssl/k8s/k8s-server.pem \
--tls-private-key-file=/apps/k8s/ssl/k8s/k8s-server-key.pem \
--kubelet-client-certificate=/apps/k8s/ssl/k8s/k8s-server.pem \
--kubelet-client-key=/apps/k8s/ssl/k8s/k8s-server-key.pem \
--service-account-key-file=/apps/k8s/ssl/k8s/k8s-ca.pem \
--requestheader-client-ca-file=/apps/k8s/ssl/k8s/k8s-ca.pem \
--proxy-client-cert-file=/apps/k8s/ssl/k8s/aggregator.pem \
--proxy-client-key-file=/apps/k8s/ssl/k8s/aggregator-key.pem \
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--service-account-signing-key-file=/apps/k8s/ssl/k8s/k8s-ca-key.pem \
--requestheader-allowed-names=aggregator \
--requestheader-group-headers=X-Remote-Group \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true \
--anonymous-auth=false \
--experimental-encryption-provider-config=/apps/k8s/config/encryptionconfig.yaml \
--enable-admissionplugins=DefaultStorageClass,DefaultTolerationSeconds,LimitRanger,NamespaceExists,Name
spaceLifecycle,NodeRestriction,PodNodeSelector,PersistentVolumeClaimResize,PodTolerat
ionRestriction,ResourceQuota,ServiceAccount,StorageObjectInUseProtection,MutatingAdmi
ssionWebhook,ValidatingAdmissionWebhook \
--disable-admission-
plugins=ExtendedResourceTolerati
plugins=ExtendedResourceToleration,ImagePolicyWebhook,LimitPodHardAntiAffinityTopolog
y,NamespaceAutoProvision,Priority,EventRateLimit,PodSecurityPolicy \
--cors-allowed-origins=.* \
--enable-swagger-ui \
--runtime-config=api/all=true \
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--authorization-mode=Node,RBAC \
--allow-privileged=true \
--apiserver-count=3 \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--default-not-ready-toleration-seconds=30 \
--default-unreachable-toleration-seconds=30 \
--audit-log-truncate-enabled \
--audit-log-path=/apps/k8s/log/api-server-audit.log \
--profiling \
--http2-max-streams-per-connection=10000 \
--event-ttl=1h \
--enable-bootstrap-token-auth=true \
--alsologtostderr=true \
--log-dir=/apps/k8s/log \
--v=2 \
--tls-ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDH
E_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES
_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 \
--endpoint-reconciler-type=lease \
--max-mutating-requests-inflight=500 \
--max-requests-inflight=1500 \
--target-ram-mb=300"
EOF
advertise-address ：apiserver 对外通告的 IP（kubernetes 服务后端节点 IP）；
default-*-toleration-seconds ：设置节点异常相关的阈值；
max-*-requests-inflight ：请求相关的最大阈值；
etcd-* ：访问 etcd 的证书和 etcd 服务器地址；
bind-address ： https 监听的 IP，不能为 127.0.0.1 ，否则外界不能访问它的安全端口 5443；
secret-port ：https 监听端口；
insecure-port=0 ：关闭监听 http 非安全端口(8080)；
tls-*-file ：指定 apiserver 使用的证书、私钥和 CA 文件；
audit-* ：配置审计策略和审计日志文件相关的参数；
client-ca-file ：验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy
等)请求所带的证书；
enable-bootstrap-token-auth ：启用 kubelet bootstrap 的 token 认证；
requestheader-* ：kube-apiserver 的 aggregator layer 相关的配置参数，proxy-client & HPA 需要使用；
requestheader-client-ca-file ：用于签名 --proxy-client-cert-file 和 --proxy-client-keyfile 指定的证书；在启用了 metric aggregator 时使用；
requestheader-allowed-names ：不能为空，值为逗号分割的 --proxy-client-cert-file 证书的 CN
名称，这里设置为 “aggregator”；
service-account-key-file ：签名 ServiceAccount Token 的公钥文件，kube-controller-manager
的 --service-account-private-key-file 指定私钥文件，两者配对使用；
runtime-config=api/all=true ： 启用所有版本的 APIs，如 autoscaling/v2alpha1；
authorization-mode=Node,RBAC 、 --anonymous-auth=false ： 开启 Node 和 RBAC 授权模式，拒绝未授权的请求；
enable-admission-plugins ：启用一些默认关闭的 plugins；
allow-privileged ：运行执行 privileged 权限的容器；
apiserver-count=3 ：指定 apiserver 实例的数量；
event-ttl ：指定 events 的保存时间；
kubelet-* ：如果指定，则使用 https 访问 kubelet APIs；需要为证书对应的用户(上面
kubernetes*.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则，否则访问 kubelet API 时提示未
授权；
proxy-client-* ：apiserver 访问 metrics-server 使用的证书；
service-cluster-ip-range ： 指定 Service Cluster IP 地址段；
service-node-port-range ： 指定 NodePort 的端口范围；
如果 kube-apiserver 机器没有运行 kube-proxy，则还需要添加 --enable-aggregator-routing=true 参数；
10.5 分发kube-apiserver 证书及配置
证书分发
# 分发server 证书
scp -r /opt/k8s/cfssl/pki/k8s/k8s-server* root@192.168.2.175:/apps/k8s/ssl/k8s
scp -r /opt/k8s/cfssl/pki/k8s/k8s-server* root@192.168.2.176:/apps/k8s/ssl/k8s
scp -r /opt/k8s/cfssl/pki/k8s/k8s-server* root@192.168.2.177:/apps/k8s/ssl/k8s
# 分发webhook证书
scp -r /opt/k8s/cfssl/pki/k8s/aggregator* root@192.168.2.175:/apps/k8s/ssl/k8s
scp -r /opt/k8s/cfssl/pki/k8s/aggregator* root@192.168.2.176:/apps/k8s/ssl/k8s
scp -r /opt/k8s/cfssl/pki/k8s/aggregator* root@192.168.2.177:/apps/k8s/ssl/k8s
配置分发
cd /opt/k8s/work
scp -r config root@192.168.2.175:/apps/k8s/
scp -r config root@192.168.2.176:/apps/k8s/
scp -r config root@192.168.2.177:/apps/k8s/
10.6 创建 kube-apiserver systemd unit 文件
k8s-master-1 k8s-master-2 k8s-master-3 节点上执行
cat > /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
Type=notify
LimitNOFILE=655350
LimitNPROC=655350
LimitCORE=infinity
LimitMEMLOCK=infinity
EnvironmentFile=-/apps/k8s/conf/kube-apiserver
ExecStart=/apps/k8s/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
10.7 启动 kube-apiserver 服务
k8s-master-1 k8s-master-2 k8s-master-3 节点上执行
# 全局刷新service
systemctl daemon-reload
# 设置kube-apiserver开机启动
systemctl enable kube-apiserver
#重启kube-apiserver
systemctl restart kube-apiserver
10.8 检查启动结果
k8s-master-1 k8s-master-2 k8s-master-3 节点上执行
systemctl status kube-apiserver|grep Active
[root@k8s-master-1 ~]# systemctl status kube-apiserver|grep Active
Active: active (running) since Fri 2022-02-11 13:49:41 CST; 3 days ago
[root@k8s-master-2 ~]# systemctl status kube-apiserver|grep Active
Active: active (running) since Fri 2022-02-11 13:49:40 CST; 3 days ago
[root@k8s-master-3 ~]# systemctl status kube-apiserver|grep Active
Active: active (running) since Mon 2022-02-14 14:39:40 CST; 1h 4min ago
10.9 验证服务状态
qist 节点上执行
部署完 kube-apiserver 集群后，在任一 qist 节点上执行如下命令：
# 配置环境变量
export KUBECONFIG=/opt/k8s/kubeconfig/admin.kubeconfig
root@Qist work# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE
ERROR
scheduler Unhealthy Get https://127.0.0.1:10259/healthz: dial tcp
127.0.0.1:10259: connect: connection refused
controller-manager Unhealthy Get https://127.0.0.1:10257/healthz: dial tcp
127.0.0.1:10257: connect: connection refused
etcd-0 Healthy {"health":"true","reason":""}
etcd-2 Healthy {"health":"true","reason":""}
etcd-1 Healthy {"health":"true","reason":""}
kubectl cluster-info
预期输出：
root@Qist work# kubectl cluster-info
Kubernetes control plane is running at https://192.168.2.175:6443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
正常输出表示集群正常
期待下次的分享，别忘了三连支持博主呀~
我是 念舒_C.ying ，期待你的关注~
附专栏链接
【云原生 · Kubernetes】runtime组件
【云原生 · Kubernetes】apiserver高可用
【云原生 · Kubernetes】kubernetes v1.23.3 二进制部署（三）
【云原生 · Kubernetes】kubernetes v1.23.3 二进制部署（二）
【云原生 · Kubernetes】kubernetes v1.23.3 二进制部署（一）
【云原生 · Kubernetes】Kubernetes 编排部署GPMall（一）
【云原生 · Kubernetes】Kubernetes容器云平台部署与运维
【云原生 · Kubernetes】部署博客系统
【云原生 · Kubernetes】部署Kubernetes集群
【云原生 · Kubernetes】Kubernetes基础环境搭建
