技术教程 7.第六篇 二进制安装 kube-apiserver 正文

7.第六篇 二进制安装 kube-apiserver

2022-11-01,,,,

文章转载自：https://mp.weixin.qq.com/s?__biz=MzI1MDgwNzQ1MQ==&mid=2247483812&idx=1&sn=e6773e5638415e627e3a351f340b7bd0&chksm=e9fdd450de8a5d468503f85abefa7a4d6f62dacd60f835722036b6958c95c8a979e77c741f01&cur_album_id=1341273083637989377&scene=189#wechat_redirect

这里会使用到以下IP，详情可以查看第二篇 环境初始化准备，以下本章节部分使用到的服务器列表，之前我们还创建了认证key， 详情查看第三篇 PKI基础概念、cfssl工具介绍及kubernetes中证书。

创建加密配置文件

v1.7+版本后新特性，kube-apiserver 将Secret数据加密后存储到etcd中，但是需要在kube-apiserver启动时配置--experimental-encryption-provider-config，这里我们定义加密配置格式如下，并且需要把这个配置文件分发到所有master服务器；

#!/bin/bash

cd /data/k8s/work

source /data/k8s/bin/env.sh

cat > encryption-config.yaml <<EOF

kind: EncryptionConfig

apiVersion: v1

resources:

  - resources:

      - secrets

    providers:

      - aescbc:

          keys:

            - name: key1

              secret: ${ENCRYPTION_KEY}

      - identity: {}

EOF

for node_ip in ${MASTER_IPS[@]}

do

  echo ">>> ${node_ip}"

  scp encryption-config.yaml root@${node_ip}:/etc/kubernetes/

done

创建审计策略文件

Kubernetes日志审计是Kube-apiserver组件的一部分，它提供了与安全相关的日志操作，日志中记录了单个用户、管理员或系统其它组件在与kube-apiserver交互请求时的全部请求处理过程。

#!/bin/bash

cd /data/k8s/work

source /data/k8s/bin/env.sh

cat > audit-policy.yaml <<EOF

apiVersion: audit.k8s.io/v1beta1

kind: Policy

rules:

  # The following requests were manually identified as high-volume and low-risk, so drop them.

  - level: None

    resources:

      - group: ""

        resources:

          - endpoints

          - services

          - services/status

    users:

      - 'system:kube-proxy'

    verbs:

      - watch

  - level: None

    resources:

      - group: ""

        resources:

          - nodes

          - nodes/status

    userGroups:

      - 'system:nodes'

    verbs:

      - get

  - level: None

    namespaces:

      - kube-system

    resources:

      - group: ""

        resources:

          - endpoints

    users:

      - 'system:kube-controller-manager'

      - 'system:kube-scheduler'

      - 'system:serviceaccount:kube-system:endpoint-controller'

    verbs:

      - get

      - update

  - level: None

    resources:

      - group: ""

        resources:

          - namespaces

          - namespaces/status

          - namespaces/finalize

    users:

      - 'system:apiserver'

    verbs:

      - get

  # Don't log HPA fetching metrics.

  - level: None

    resources:

      - group: metrics.k8s.io

    users:

      - 'system:kube-controller-manager'

    verbs:

      - get

      - list

  # Don't log these read-only URLs.

  - level: None

    nonResourceURLs:

      - '/healthz*'

      - /version

      - '/swagger*'

  # Don't log events requests.

  - level: None

    resources:

      - group: ""

        resources:

          - events

  # node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes

  - level: Request

    omitStages:

      - RequestReceived

    resources:

      - group: ""

        resources:

          - nodes/status

          - pods/status

    users:

      - kubelet

      - 'system:node-problem-detector'

      - 'system:serviceaccount:kube-system:node-problem-detector'

    verbs:

      - update

      - patch

  - level: Request

    omitStages:

      - RequestReceived

    resources:

      - group: ""

        resources:

          - nodes/status

          - pods/status

    userGroups:

      - 'system:nodes'

    verbs:

      - update

      - patch

  # deletecollection calls can be large, don't log responses for expected namespace deletions

  - level: Request

    omitStages:

      - RequestReceived

    users:

      - 'system:serviceaccount:kube-system:namespace-controller'

    verbs:

      - deletecollection

  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,

  # so only log at the Metadata level.

  - level: Metadata

    omitStages:

      - RequestReceived

    resources:

      - group: ""

        resources:

          - secrets

          - configmaps

      - group: authentication.k8s.io

        resources:

          - tokenreviews

  # Get repsonses can be large; skip them.

  - level: Request

    omitStages:

      - RequestReceived

    resources:

      - group: ""

      - group: admissionregistration.k8s.io

      - group: apiextensions.k8s.io

      - group: apiregistration.k8s.io

      - group: apps

      - group: authentication.k8s.io

      - group: authorization.k8s.io

      - group: autoscaling

      - group: batch

      - group: certificates.k8s.io

      - group: extensions

      - group: metrics.k8s.io

      - group: networking.k8s.io

      - group: policy

      - group: rbac.authorization.k8s.io

      - group: scheduling.k8s.io

      - group: settings.k8s.io

      - group: storage.k8s.io

    verbs:

      - get

      - list

      - watch

  # Default level for known APIs

  - level: RequestResponse

    omitStages:

      - RequestReceived

    resources:

      - group: ""

      - group: admissionregistration.k8s.io

      - group: apiextensions.k8s.io

      - group: apiregistration.k8s.io

      - group: apps

      - group: authentication.k8s.io

      - group: authorization.k8s.io

      - group: autoscaling

      - group: batch

      - group: certificates.k8s.io

      - group: extensions

      - group: metrics.k8s.io

      - group: networking.k8s.io

      - group: policy

      - group: rbac.authorization.k8s.io

      - group: scheduling.k8s.io

      - group: settings.k8s.io

      - group: storage.k8s.io

  # Default level for all other requests.

  - level: Metadata

    omitStages:

      - RequestReceived

EOF

# 分发审计策略文件

for node_ip in ${MASTER_IPS[@]}

do

    echo ">>> ${node_ip}"

    scp audit-policy.yaml root@${node_ip}:/etc/kubernetes/audit-policy.yaml

done

创建kube-apiserver启动文件

创建启动文件模板脚本如下

#!/bin/bash

cd /data/k8s/work/

source /data/k8s/bin/env.sh

cat > kube-apiserver.service.template <<EOF

[Unit]

Description=Kubernetes API Server

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=network.target

[Service]

WorkingDirectory=/data/k8s/k8s/kube-apiserver

ExecStart=/data/k8s/bin/kube-apiserver \

  --advertise-address=##NODE_IP## \

  --default-not-ready-toleration-seconds=360 \

  --default-unreachable-toleration-seconds=360 \

  --feature-gates=DynamicAuditing=true \

  --max-mutating-requests-inflight=2000 \

  --max-requests-inflight=4000 \

  --default-watch-cache-size=200 \

  --delete-collection-workers=2 \

  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml \

  --etcd-cafile=/etc/kubernetes/cert/ca.pem \

  --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \

  --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \

  --etcd-servers=https://etcd01.k8s.vip:2379,https://etcd02.k8s.vip:2379,https://etcd03.k8s.vip:2379 \

  --bind-address=##NODE_IP## \

  --secure-port=6443 \

  --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \

  --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \

  --insecure-port=0 \

  --audit-dynamic-configuration \

  --audit-log-maxage=15 \

  --audit-log-maxbackup=3 \

  --audit-log-maxsize=100 \

  --audit-log-truncate-enabled \

  --audit-log-path=/data/k8s/k8s/kube-apiserver/audit.log \

  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \

  --profiling \

  --anonymous-auth=false \

  --client-ca-file=/etc/kubernetes/cert/ca.pem \

  --enable-bootstrap-token-auth \

  --requestheader-allowed-names="aggregator" \

  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \

  --requestheader-extra-headers-prefix="X-Remote-Extra-" \

  --requestheader-group-headers=X-Remote-Group \

  --requestheader-username-headers=X-Remote-User \

  --service-account-key-file=/etc/kubernetes/cert/ca.pem \

  --authorization-mode=Node,RBAC \

  --runtime-config=api/all=true \

  --enable-admission-plugins=NodeRestriction \

  --allow-privileged=true \

  --apiserver-count=3 \

  --event-ttl=168h \

  --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \

  --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \

  --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \

  --kubelet-https=true \

  --kubelet-timeout=10s \

  --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \

  --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \

  --service-cluster-ip-range=10.254.0.0/16 \

  --service-node-port-range=1024-32767 \

  --logtostderr=true \

  --v=2

Restart=on-failure

RestartSec=10

Type=notify

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

EOF

配置详解如下

为各节点创建并分发启动配置文件

#!/bin/bash

cd /data/k8s/work

source /data/k8s/bin/env.sh

# 由模板替换成各服务器启动文件

for (( i=0; i < 3; i++ ))

do

    sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${NODE_IPS[i]}.service

done

# 分发到所有的master节点服务器

for node_ip in ${MASTER_IPS[@]}

do

    echo ">>> ${node_ip}"

    scp kube-apiserver-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-apiserver.service

done

远程启动kube-apiserver服务

创建审计日志目录并启动服务

#!/bin/bash

source /data/k8s/bin/env.sh

for node_ip in ${MASTER_IPS[@]}

do

    echo ">>> ${node_ip}"

    ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"

    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"

done

检查服务启动情况

#!/bin/bash

source /data/k8s/bin/env.sh

for node_ip in ${MASTER_IPS[@]}

do

    echo ">>> ${node_ip}"

    ssh root@${node_ip} "systemctl status kube-apiserver |grep -i 'Active:'"

done

结果如下：

>>> 192.168.16.104

   Active: active (running) since Sun 2019-12-29 18:03:55 CST; 2 weeks 0 days ago

>>> 192.168.16.105

   Active: active (running) since Sat 2019-12-28 19:59:48 CST; 2 weeks 1 days ago

>>> 192.168.16.106

   Active: active (running) since Sat 2019-12-28 19:59:53 CST; 2 weeks 1 days ago

总结

kubernetes v 1.17.0二进制安装过程比较简单，从中能够学习到很多知识点，尤其是之前不熟悉的参数，经过查询资料有了一个更深层次的认知。

参考链接：https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/

7.第六篇 二进制安装 kube-apiserver的相关教程结束。

《7.第六篇 二进制安装 kube-apiserver.doc》

下载本文的Word格式文档，以方便收藏与打印。

Copyright 2023. 昆居客 - IT编程,互联网信息技术文档大全!