主要参考https://github.com/opsnull/follow-me-install-kubernetes-cluster，采用Flanel和docker
系统信息
| 角色 | 系统 | CPU Core | 内存 | 主机名称 | ip | 安装组件 |
|---|---|---|---|---|---|---|
| master | 18.04.1-Ubuntu | 4 | 8G | master | 192.168.0.107 | kubectl,kube-apiserver,kube-controller-manager,kube-scheduler,etcd,flannald |
| slave | 18.04.1-Ubuntu | 4 | 4G | slave | 192.168.0.114 | docker,flannald,kubelet,kube-proxy,coredns |
k8s&docker版本
| 软件 | 版本 |
|---|---|
| k8s | 1.17.2 |
| etcd | v3.3.18 |
| coredns | 1.6.6(docker镜像） |
| Flanel | v0.11.0 |
| docker | 18.09 |
安装前准备（主节点和从节点都需要执行）
关闭swap
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
配置常用软件安装源
在/etc/apt/sources.list.d/ 追加system.list文件,内容如下
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted
deb http://mirrors.aliyun.com/ubuntu/ bionic universe
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates universe
deb http://mirrors.aliyun.com/ubuntu/ bionic multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
执行
sudo apt-get update
创建工作目录
mkdir -p /opt/k8s/{bin,work} /etc/{kubernetes,etcd}/cert
将 /opt/k8s/bin追加到$PATH中
echo 'PATH=/opt/k8s/bin:$PATH' >>/root/.bashrc
source /root/.bashrc
安装ssh服务，并设置root可以执行
apt install openssh-server
#编辑/etc/ssh/sshd_config文件，在#PermitRootLogin prohibit-password下追加PermitRootLogin yes ，重启ssh服务
systemctl restart ssh.service
安装依赖工具包
apt install -y ipvsadm ipset curl jq
设置主机名
cat >> /etc/hosts <<EOF
192.168.0.107 master
192.168.0.114 slave
EOF
添加节点信任关系,只用在master节点上执行
ssh-keygen -t rsa
ssh-copy-id root@192.168.0.114
创建CA根证书和秘钥(在master节点上执行）
安装cfssl工具集
cd /opt/k8s/work
wget https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64
cp cfssl_1.4.1_linux_amd64 /opt/k8s/bin/cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64
cp cfssljson_1.4.1_linux_amd64 /opt/k8s/bin/cfssljson
wget https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl-certinfo_1.4.1_linux_amd64
cp cfssl-certinfo_1.4.1_linux_amd64 /opt/k8s/bin/cfssl-certinfo
chmod +x /opt/k8s/bin/*
创建CA配置文件
cd /opt/k8s/work
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
signing：表示该证书可用于签名其它证书（生成的 ca.pem 证书中 CA=TRUE）；
server auth：表示 client 可以用该该证书对 server 提供的证书进行验证；
client auth：表示 server 可以用该该证书对 client 提供的证书进行验证；
expiry : "87600h"：证书有效期设置为 10 年；
创建证书签名请求文件
cd /opt/k8s/work
cat > ca-csr.json <<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "NanJing",
"L": "NanJing",
"O": "k8s",
"OU": "system"
}
],
"ca": {
"expiry": "87600h"
}
}
EOF
生成证书
cd /opt/k8s/work
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls ca*
安装证书
cd /opt/k8s/work
cp ca*.pem ca-config.json /etc/kubernetes/cert
# 分发到从节点
export node_ip=192.168.0.114
scp ca*.pem ca-config.json root@${node_ip}:/etc/kubernetes/cert/
部署 etcd（在master节点上执行）
下载安装etcd
cd /opt/k8s/work
wget https://github.com/etcd-io/etcd/releases/download/v3.3.18/etcd-v3.3.18-linux-amd64.tar.gz
tar -xvf etcd-v3.3.18-linux-amd64.tar.gz
安装etcd
cd /opt/k8s/work
cp etcd-v3.3.18-linux-amd64/etcd* /opt/k8s/bin/
chmod +x /opt/k8s/bin/*
创建 etcd 证书和私钥
创建证书签名请求文件
cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.0.107"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "NanJing",
"L": "NanJing",
"O": "k8s",
"OU": "system"
}
]
}
EOF
指定授权使用该证书的 etcd 节点 IP 列表
生成证书和私钥
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ls etcd*pem
安装证书
cd /opt/k8s/work
cp etcd*.pem /etc/etcd/cert/
创建etcd启动文件
cat> /etc/systemd/system/etcd.service<< EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/data/k8s/etcd/data
ExecStart=/opt/k8s/bin/etcd \\
--data-dir=/etc/etcd/cfg/etcd \\
--name=etcd-chengf \\
--cert-file=/etc/etcd/cert/etcd.pem \\
--key-file=/etc/etcd/cert/etcd-key.pem \\
--trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
--peer-cert-file=/etc/etcd/cert/etcd.pem \\
--peer-key-file=/etc/etcd/cert/etcd-key.pem \\
--peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\
--peer-client-cert-auth \\
--client-cert-auth \\
--listen-peer-urls=https://192.168.0.107:2380 \\
--initial-advertise-peer-urls=https://192.168.0.107:2380 \\
--listen-client-urls=https://192.168.0.107:2379,http://127.0.0.1:2379 \\
--advertise-client-urls=https://192.168.0.107:2379 \\
--initial-cluster-token=etcd-cluster-0\\
--initial-cluster=etcd-chengf=https://192.168.0.107:2380 \\
--initial-cluster-state=new \\
--auto-compaction-mode=periodic \\
--auto-compaction-retention=1 \\
--max-request-bytes=33554432 \\
--quota-backend-bytes=6442450944 \\
--heartbeat-interval=250 \\
--election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
WorkingDirectory、--data-dir：指定工作目录和数据目录，需在启动服务前创建这个目录；
--name：指定节点名称，当 --initial-cluster-state 值为 new 时，--name 的参数值必须位于 --initial-cluster 列表中；
--cert-file、--key-file：etcd server 与 client 通信时使用的证书和私钥；
--trusted-ca-file：签名 client 证书的 CA 证书，用于验证 client 证书；
--peer-cert-file、--peer-key-file：etcd 与 peer 通信使用的证书和私钥；
--peer-trusted-ca-file：签名 peer 证书的 CA 证书，用于验证 peer 证书；
创建etcd数据目录
mkdir -p /data/k8s/etcd/data
启动 etcd 服务
systemctl enable etcd && systemctl start etcd
检查启动结果
systemctl status etcd|grep Active
确保状态为 active (running)，否则查看日志，确认原因
如果出现异常，通过如下命令查看
journalctl -u etcd
验证服务状态
export ETCD_ENDPOINTS=https://192.168.0.107:2379
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/etcd/cert/etcd.pem \
--key-file=/etc/etcd/cert/etcd-key.pem cluster-health
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/etcd/cert/etcd.pem \
--key-file=/etc/etcd/cert/etcd-key.pem member list
输出结果
root@master:/opt/k8s/work# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem cluster-health
member c0d3b56a9878e38f is healthy: got healthy result from https://192.168.0.107:2379
cluster is healthy
root@master:/opt/k8s/work# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pemmember list
c0d3b56a9878e38f: name=etcd-chengf peerURLs=https://192.168.0.107:2380 clientURLs=https://192.168.0.107:2379 isLeader=true
```
部署 flannel 网络(在master节点上执行）
kubernetes组件kubelet服务依赖docker服务，docker网络需要用flannel来配置docker0网桥的ip地址，所以需要先安装flannel网络组建
flannel 使用 vxlan 技术为各节点创建一个可以互通的 Pod 网络，使用的端口为 UDP 8472（需要开放该端口，如公有云 AWS 等）。
flanneld 第一次启动时，从 etcd 获取配置的 Pod 网段信息，为本节点分配一个未使用的地址段，然后创建 flannedl.1 网络接口（也可能是其它名称，如 flannel1 等）。
flannel 将分配给自己的 Pod 网段信息写入 /run/flannel/docker 文件，docker 后续使用这个文件中的环境变量设置 docker0 网桥，从而从这个地址段为本节点的所有 Pod 容器分配 IP
下载和安装flanneld 二进制文件
cd /opt/k8s/work
mkdir flannel
wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel
cp flannel/{flanneld,mk-docker-opts.sh} /opt/k8s/bin/
export node_ip=192.168.0.114
scp flannel/{flanneld,mk-docker-opts.sh} root@${192.168.0.114}:/opt/k8s/bin/
创建 flanneld 证书和私钥
flanneld 从 etcd 集群存取网段分配信息，而 etcd 集群启用了双向 x509 证书认证，所以需要为 flanneld 生成证书和私钥。
创建证书签名请求
cd /opt/k8s/work
cat > flanneld-csr.json <<EOF
{
"CN": "flanneld",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "NanJing",
"L": "NanJing",
"O": "k8s",
"OU": "system"
}
]
}
EOF
生成证书和私钥
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
ls flanneld*pem
将生成的证书和私钥分发到所有节点
cd /opt/k8s/work
mkdir -p /etc/flanneld/cert
cp flanneld*.pem /etc/flanneld/cert
export node_ip=192.168.0.114
ssh root@${node_ip} "mkdir -p /etc/flanneld/cert"
scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert
向 etcd 写入集群 Pod 网段信息
cd /opt/k8s/work
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
export ETCD_ENDPOINTS="https://192.168.0.107:2379"
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/opt/k8s/work/ca.pem \
--cert-file=/opt/k8s/work/flanneld.pem \
--key-file=/opt/k8s/work/flanneld-key.pem \
mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
写入的 Pod 网段 Network 网络段对应的数值（如 /16）必须小于 SubnetLen对应的值（如24）
创建 flanneld 服务的启动文件
cd /opt/k8s/work
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
export ETCD_ENDPOINTS="https://192.168.0.107:2379"
cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
-etcd-cafile=/etc/kubernetes/cert/ca.pem \\
-etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
-etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
-etcd-endpoints=${ETCD_ENDPOINTS} \\
-etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
-ip-masq
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网段信息，通过-d参数写入 /run/flannel/docker 文件，后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥， -k 参数控制生成文件中变量的名称，下面docker启动时会用到这个变量；
flanneld 使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口（如内网和公网）的节点，可以用 -iface 参数指定通信接口;
-ip-masq: flanneld 为访问 Pod 网络外的流量设置 SNAT 规则，同时将传递给 Docker 的变量 --ip-masq（/run/flannel/docker 文件中）设置为 false，这样 Docker 将不再创建 SNAT 规则； Docker 的 --ip-masq 为 true 时，创建的 SNAT 规则比较“暴力”：将所有本节点 Pod 发起的、访问非 docker0 接口的请求做 SNAT，这样访问其他节点 Pod 的请求来源 IP 会被设置为 flannel.1 接口的 IP，导致目的 Pod 看不到真实的来源 Pod IP。 flanneld 创建的 SNAT 规则比较温和，只对访问非 Pod 网段的请求做 SNAT
分发flanneld服务
cd /opt/k8s/work
cp flanneld.service /etc/systemd/system/
export node_ip=192.168.0.114
scp flanneld.service root@${node_ip}:/etc/systemd/system/
启动flanneld服务
systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld
ssh root@${node_ip) "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"
检查启动结果
systemctl status flanneld|grep Active
export node_ip=192.168.0.114
ssh root@${node_ip} "systemctl status flanneld|grep Active"
确保状态为 active (running)，否则查看日志，确认原因
如果出现异常，通过如下命令查看
journalctl -u flanneld
检查分配给各 flanneld 的 Pod 网段信息
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
export ETCD_ENDPOINTS="https://192.168.0.107:2379"
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
get ${FLANNEL_ETCD_PREFIX}/config
输出结果
{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}
查看已分配的 Pod 子网段列表
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
export ETCD_ENDPOINTS="https://192.168.0.107:2379"
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
ls ${FLANNEL_ETCD_PREFIX}/subnets
输出结果
/kubernetes/network/subnets/172.30.22.0-24
/kubernetes/network/subnets/172.30.78.0-24
检查节点 flannel 网络信息
root@master:/opt/k8s/work# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether 04:92:26:13:92:2b brd ff:ff:ff:ff:ff:ff
3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d0:c5:d3:57:73:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.107/24 brd 192.168.0.255 scope global dynamic noprefixroute wlp3s0
valid_lft 6385sec preferred_lft 6385sec
inet6 fe80::1fda:e90a:207a:67e4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 12:cb:66:43:de:36 brd ff:ff:ff:ff:ff:ff
inet 172.30.22.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::10cb:66ff:fe43:de36/64 scope link
valid_lft forever preferred_lft forever
root@master:/opt/k8s/work# ip route show |grep flannel.1
172.30.78.0/24 via 172.30.78.0 dev flannel.1 onlink
验证各节点能通过 Pod 网段互通
root@master:/opt/k8s/work# ip addr show flannel.1 |grep -w inet
inet 172.30.22.0/32 scope global flannel.1
root@master:/opt/k8s/work# ssh 192.168.0.114 "/sbin/ip addr show flannel.1|grep -w inet"
inet 172.30.78.0/32 scope global flannel.1
root@master:/opt/k8s/work# ping -c 1 172.30.78.0
PING 172.30.78.0 (172.30.78.0) 56(84) bytes of data.
64 bytes from 172.30.78.0: icmp_seq=1 ttl=64 time=80.7 ms
--- 172.30.78.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 80.707/80.707/80.707/0.000 ms
root@master:/opt/k8s/work# ssh 192.168.0.114 "ping -c 1 172.30.22.0"
PING 172.30.22.0 (172.30.22.0) 56(84) bytes of data.
64 bytes from 172.30.22.0: icmp_seq=1 ttl=64 time=4.09 ms
--- 172.30.22.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.094/4.094/4.094/0.000 ms
生成文件
root@master:/opt/k8s/work# cat /run/flannel/subnet.env
FLANNEL_NETWORK=172.30.0.0/16
FLANNEL_SUBNET=172.30.22.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
root@master:/opt/k8s/work# cat /run/flannel/docker
DOCKER_OPT_BIP="--bip=172.30.22.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.30.22.1/24 --ip-masq=false --mtu=1450"
部署docker服务（在master节点上执行）
下载和分发 docker 二进制文件
cd /opt/k8s/work
wget https://download.docker.com/linux/static/stable/x86_64/docker-18.09.6.tgz
tar -xvf docker-18.09.6.tgz
分发二进制文件到所有 worker 节点
cd /opt/k8s/work
export node_ip=192.168.0.114
scp docker/* root@${node_ip}:/opt/k8s/bin/
ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
创建docker服务启动文件
cd /opt/k8s/work
cat > docker.service <<"EOF"
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
[Service]
WorkingDirectory=/data/k8s/docker
Environment="PATH=/opt/k8s/bin:/bin:/sbin:/usr/bin:/usr/sbin"
EnvironmentFile=-/run/flannel/docker
ExecStart=/opt/k8s/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
EOF 前后有双引号，这样 bash 不会替换文档中的变量，如 $DOCKER_NETWORK_OPTIONS (这些环境变量是 systemd 负责替换的。)；
dockerd 运行时会调用其它 docker 命令，如 docker-proxy，所以需要将 docker 命令所在的目录加到 PATH 环境变量中；
flanneld 启动时将网络配置写入 /run/flannel/docker 文件中，dockerd 启动前读取该文件中的环境变量 DOCKER_NETWORK_OPTIONS ，然后设置 docker0 网桥网段；
docker 从 1.13 版本开始，可能将 iptables FORWARD chain的默认策略设置为DROP，从而导致 ping 其它 Node 上的 Pod IP 失败，遇到这种情况时，需要手动设置策略为 ACCEPT：
export node_ip=192.168.0.114
ssh root@${node_ip} "/sbin/iptables -P FORWARD ACCEPT"
分发 docker.service 文件到所有 worker 机器:
cd /opt/k8s/work
export node_ip=192.168.0.114
scp docker.service root@${node_ip}:/etc/systemd/system/
配置和分发 docker 配置文件
使用国内的仓库镜像服务器以加快 pull image 的速度，同时增加下载的并发数 (需要重启 dockerd 生效)：
cd /opt/k8s/work
cat > docker-daemon.json <<EOF
{
"registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"],
"max-concurrent-downloads": 20,
"live-restore": true,
"max-concurrent-uploads": 10,
"data-root": "/data/k8s/docker/data",
"log-opts": {
"max-size": "100m",
"max-file": "5"
}
}
EOF
分发 docker 配置文件到所有 worker 节点：
cd /opt/k8s/work
export node_ip=192.168.0.114
ssh root@${node_ip} "mkdir -p /etc/docker/ /data/k8s/docker/data"
scp docker-daemon.json root@${node_ip}:/etc/docker/daemon.json
启动 docker 服务
export node_ip=192.168.0.114
ssh root@${node_ip} "systemctl daemon-reload && systemctl enable docker && systemctl restart docker"
检查服务运行状态
export node_ip=192.168.0.114
ssh root@${node_ip} "systemctl status docker|grep Active"
确保状态为 active (running)，否则查看日志，确认原因
如果出现异常，通过如下命令查看
journalctl -u docker
检查 docker0 网桥
export node_ip=192.168.0.114
ssh root@${node_ip} "/sbin/ip addr show flannel.1 && /sbin/ip addr show docker0"
确认各 worker 节点的 docker0 网桥和 flannel.1 接口的 IP 处于同一个网段中
输出内容
export node_ip=192.168.0.114
root@master:/opt/k8s/work# ssh root@${node_ip} "/sbin/ip addr show flannel.1 && /sbin/ip addr show docker0"
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether f2:fc:0f:7e:98:e4 brd ff:ff:ff:ff:ff:ff
inet 172.30.78.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::f0fc:fff:fe7e:98e4/64 scope link
valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:fd:1f:8f:d8 brd ff:ff:ff:ff:ff:ff
inet 172.30.78.1/24 brd 172.30.78.255 scope global docker0
valid_lft forever preferred_lft forever
注意: 如果您的服务安装顺序不对或者机器环境比较复杂, docker服务早于flanneld服务安装，此时 worker 节点的 docker0 网桥和 flannel.1 接口的 IP可能不会同处同一个网段下，这个时候请先停止docker服务, 手工删除docker0网卡，重新启动docker服务后即可修复
systemctl stop docker
ip link delete docker0
systemctl start docker
查看 docker 的状态信息
root@slave:/opt/k8s/work# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.09.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.0.0-23-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.741GiB
Name: slave
ID: IDMG:7A6F:UNTP:IWVM:ZBK5:VHJ4:STC5:UXZX:HQT6:UUNE:YDOC:I27L
Docker Root Dir: /data/k8s/docker/data
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://docker.mirrors.ustc.edu.cn/
https://hub-mirror.c.163.com/
Live Restore Enabled: true
Product License: Community Engine
WARNING: No swap limit support
部署 master 节点（在master节点上执行）
下载最新版本二进制文件
cd /opt/k8s/work
wget https://dl.k8s.io/v1.17.2/kubernetes-server-linux-amd64.tar.gz # 目前国内不能直接下载，需翻墙
tar -xzvf kubernetes-server-linux-amd64.tar
安装对应的k8s命令
cd /opt/k8s/work
cp kubernetes/server/bin/{apiextensions-apiserver,kubeadm,kube-apiserver,kube-controller-manager,kubectl,kubelet,kube-proxy,kube-scheduler,mounter} /opt/k8s/bin/
#将kubelet、kube-proxy分发到worker节点
export node_ip=192.168.0.114
scp kubernetes/server/bin/{kubelet,kube-proxy} root@${node_ip}:/opt/k8s/bin/
配置kubectl
kubectl 使用 https 协议与 kube-apiserver 进行安全通信，kube-apiserver 对 kubectl 请求包含的证书进行认证和授权。
kubectl 后续用于集群管理，所以这里创建具有最高权限的 admin 证书。
创建 admin 证书和私钥
创建证书签名请求文件
cd /opt/k8s/work
cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "NanJing",
"L": "NanJing",
"O": "system:masters",
"OU": "system"
}
]
}
EOF
O: system:masters：kube-apiserver 收到使用该证书的客户端请求后，为请求添加组（Group）认证标识 system:masters；
预定义的 ClusterRoleBinding cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予操作集群所需的最高权限；
该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空；
生成证书和私钥
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes admin-csr.json | cfssljson -bare admin
ls admin*
安装证书
cd /opt/k8s/work
cp admin*.pem /etc/kubernetes/cert
创建 kubeconfig 文件
cd /opt/k8s/work
export KUBE_APISERVER=https://192.168.0.107:6443
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/cert/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kubectl.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials admin \
--client-certificate=/etc/kubernetes/cert/admin.pem \
--client-key=/etc/kubernetes/cert/admin-key.pem \
--embed-certs=true \
--kubeconfig=kubectl.kubeconfig
# 设置上下文参数
kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin \
--kubeconfig=kubectl.kubeconfig
# 设置默认上下文
kubectl config use-context kubernetes --kubeconfig=kubectl.kubeconfig
--certificate-authority：验证 kube-apiserver 证书的根证书；
--client-certificate、--client-key：刚生成的 admin 证书和私钥，与 kube-apiserver https 通信时使用；
--embed-certs=true：将 ca.pem 和 admin.pem 证书内容嵌入到生成的 kubectl.kubeconfig 文件中；
--server：指定 kube-apiserver 的地址；
分发 kubeconfig 文件（其他用户想要访问kubernetes时，也需要把此文件copy到对应的用户目录）
cd /opt/k8s/work
mkdir -p ~/.kube
cp kubectl.kubeconfig ~/.kube/config
配置kubectl自动补全功能
root@master:/opt/k8s/work# apt install -y bash-completion
root@master:/opt/k8s/work# locate bash_completion /usr/share/bash-completion/bash_completion
root@master:/opt/k8s/work# source /usr/share/bash-completion/bash_completion
root@master:/opt/k8s/work# source <(kubectl completion bash)
root@master:/opt/k8s/work# echo 'source <(kubectl completion bash)' >>~/.bashrc
配置kube-apiserver
创建 kubernetes-api 证书和私钥
创建证书签名请求文件
cd /opt/k8s/work
cat > kubernetes-csr.json <<EOF
{
"CN": "kubernetes-api",
"hosts": [
"127.0.0.1",
"192.168.0.107",
"10.254.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local."
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "NanJing",
"L": "NanJing",
"O": "k8s",
"OU": "system"
}
]
}
EOF
生成证书和私钥
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
ls kubernetes*
安装证书
cd /opt/k8s/work
cp kubernetes*.pem /etc/kubernetes/cert/
创建kube-api服务启动文件
export ETCD_ENDPOINTS="https://192.168.0.107:2379"
export SERVICE_CIDR="10.254.0.0/16"
export NODE_PORT_RANGE=80-60000
cat > /etc/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=/data/k8s/k8s/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \\
--advertise-address=192.168.0.107 \\
--etcd-cafile=/etc/kubernetes/cert/ca.pem \\
--etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
--etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
--etcd-servers=${ETCD_ENDPOINTS} \\
--bind-address=192.168.0.107 \\
--secure-port=6443 \\
--tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
--tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
--audit-log-maxage=15 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-truncate-enabled \\
--audit-log-path=/data/k8s/k8s/kube-apiserver/audit.log \\
--profiling \\
--anonymous-auth=false \\
--client-ca-file=/etc/kubernetes/cert/ca.pem \\
--enable-bootstrap-token-auth \\
--service-account-key-file=/etc/kubernetes/cert/ca-key.pem \\
--authorization-mode=Node,RBAC \\
--runtime-config=api/all=true \\
--allow-privileged=true \\
--event-ttl=168h \\
--kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\
--kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
--kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
--kubelet-https=true \\
--kubelet-timeout=10s \\
--service-cluster-ip-range=${SERVICE_CIDR} \\
--service-node-port-range=${NODE_PORT_RANGE} \\
--logtostderr=true \\
--v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
创建kube-api工作目录
mkdir -p /data/k8s/k8s/kube-apiserver
启动 kube-apiserver 服务
systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver
检查启动结果
systemctl status kube-apiserver |grep Active
确保状态为 active (running)，否则查看日志，确认原因
如果出现异常，通过如下命令查看
journalctl -u kube-apiserver
检查 kube-apiserver 运行状态
root@master:/opt/k8s/work# kubectl cluster-info
Kubernetes master is running at https://192.168.0.107:6443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
root@master:/opt/k8s/work# kubectl get all --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 2m30s
root@master:/opt/k8s/work# kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
etcd-0 Healthy {"health":"true"}
配置kube-controller-manager
创建 kube-controller-manager 证书和私钥
创建证书签名请求文件
cd /opt/k8s/work
cat > kube-controller-manager-csr.json <<EOF
{
"CN": "system:kube-controller-manager",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"127.0.0.1",
"192.168.0.107"
],
"names": [
{
"C": "CN",
"ST": "NanJing",
"L": "NanJing",
"O": "system:kube-controller-manager",
"OU": "system"
}
]
}
EOF
CN 和 O 均为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。
生成证书和私钥
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
ls kube-controller-manager*pem
安装证书
cd /opt/k8s/work
cp kube-controller-manager*.pem /etc/kubernetes/cert/
创建 kubeconfig 文件
kube-controller-manager 使用此文件访问apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-controller-manager 证书等信息
cd /opt/k8s/work
export KUBE_APISERVER=https://192.168.0.107:6443
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/work/ca.pem \
--embed-certs=true \
--server="${KUBE_APISERVER}" \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \
--client-certificate=kube-controller-manager.pem \
--client-key=kube-controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context system:kube-controller-manager \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
分发 kubeconfig
cd /opt/k8s/work
cp kube-controller-manager.kubeconfig /etc/kubernetes/kube-controller-manager.kubeconfig
创建kube-controller-manager服务启动文件
export SERVICE_CIDR="10.254.0.0/16"
cat > /etc/systemd/system/kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory=/data/k8s/k8s/kube-controller-manager
ExecStart=/opt/k8s/bin/kube-controller-manager \\
--profiling \\
--cluster-name=kubernetes \\
--kube-api-qps=1000 \\
--kube-api-burst=2000 \\
--leader-elect \\
--use-service-account-credentials\\
--concurrent-service-syncs=2 \\
--bind-address=192.168.0.107 \\
--secure-port=10252 \\
--tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\
--tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\
--port=0 \\
--authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--client-ca-file=/etc/kubernetes/cert/ca.pem \\
--authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\
--cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\
--experimental-cluster-signing-duration=87600h \\
--horizontal-pod-autoscaler-sync-period=10s \\
--concurrent-deployment-syncs=10 \\
--concurrent-gc-syncs=30 \\
--node-cidr-mask-size=24 \\
--service-cluster-ip-range=${SERVICE_CIDR} \\
--pod-eviction-timeout=6m \\
--terminated-pod-gc-threshold=10000 \\
--root-ca-file=/etc/kubernetes/cert/ca.pem \\
--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\
--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\
--logtostderr=true \\
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
创建kube-controller-manager工作目录
mkdir -p /data/k8s/k8s/kube-controller-manager
启动 kube-controller-manager服务
systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager
检查启动结果
systemctl status kube-controller-manager |grep Active
确保状态为 active (running)，否则查看日志，确认原因
如果出现异常，通过如下命令查看
journalctl -u kube-controller-manager
检查 kube-controller-manager 运行状态
root@master:/opt/k8s/work# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"master_6e2dfb91-8eaa-42d0-ba83-be669b99801f","leaseDurationSeconds":15,"acquireTime":"2020-02-09T13:37:08Z","renewTime":"2020-02-09T13:38:02Z","leaderTransitions":0}'
creationTimestamp: "2020-02-09T13:37:08Z"
name: kube-controller-manager
namespace: kube-system
resourceVersion: "888"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
uid: 5aa2c4a1-5ded-4870-900e-63dfd212c912
root@master:/opt/k8s/work# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://192.168.0.107:10252/healthz
ok
配置kube-scheduler
创建 kube-scheduler 证书和私钥
创建证书签名请求文件
cd /opt/k8s/work
cat > kube-scheduler-csr.json <<EOF
{
"CN": "system:kube-scheduler",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"127.0.0.1",
"192.168.0.107"
],
"names": [
{
"C": "CN",
"ST": "NanJing",
"L": "NanJing",
"O": "system:kube-scheduler",
"OU": "system"
}
]
}
EOF
CN 和 O 均为 system:kube-scheduler，kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 赋予 kube-scheduler 工作所需的权限。
生成证书和私钥
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
ls kube-scheduler*pem
安装证书
cd /opt/k8s/work
cp kube-scheduler*.pem /etc/kubernetes/cert/
创建 kubeconfig 文件
kube-scheduler 使用此文件访问apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-scheduler证书等信息
cd /opt/k8s/work
export KUBE_APISERVER=https://192.168.0.107:6443
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/work/ca.pem \
--embed-certs=true \
--server="${KUBE_APISERVER}" \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \
--client-certificate=kube-scheduler.pem \
--client-key=kube-scheduler-key.pem \
--embed-certs=true \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context system:kube-scheduler \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
分发 kubeconfig
cd /opt/k8s/work
cp kube-scheduler.kubeconfig /etc/kubernetes/kube-scheduler.kubeconfig
创建 kube-scheduler 配置文件
cd /opt/k8s/work
cat >kube-scheduler.yaml <<EOF
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
bindTimeoutSeconds: 600
clientConnection:
burst: 200
kubeconfig: "/etc/kubernetes/kube-scheduler.kubeconfig"
qps: 100
enableContentionProfiling: false
enableProfiling: true
hardPodAffinitySymmetricWeight: 1
healthzBindAddress: 192.168.0.107:10251
leaderElection:
leaderElect: true
metricsBindAddress: 192.168.0.107:10251
EOF
cp kube-scheduler.yaml /etc/kubernetes/kube-scheduler.yaml
创建kube-scheduler服务启动文件
cat > /etc/systemd/system/kube-scheduler.service <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory=/data/k8s/k8s/kube-scheduler
ExecStart=/opt/k8s/bin/kube-scheduler \\
--config=/etc/kubernetes/kube-scheduler.yaml \\
--bind-address=192.168.0.107 \\
--secure-port=10259 \\
--port=0 \\
--tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \\
--tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem \\
--authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\
--client-ca-file=/etc/kubernetes/cert/ca.pem \\
--authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\
--logtostderr=true \\
--v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF
创建kube-scheduler工作目录
mkdir -p /data/k8s/k8s/kube-scheduler
启动 kube-scheduler服务
systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler
检查启动结果
systemctl status kube-scheduler |grep Active
确保状态为 active (running)，否则查看日志，确认原因
如果出现异常，通过如下命令查看
journalctl -u kube-scheduler
检查 kube-scheduler 运行状态
root@master:/opt/k8s/work# kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"master_383054c4-58d8-4c24-a766-551a92492219","leaseDurationSeconds":15,"acquireTime":"2020-02-10T02:17:40Z","renewTime":"2020-02-10T02:18:09Z","leaderTransitions":0}'
creationTimestamp: "2020-02-10T02:17:41Z"
name: kube-scheduler
namespace: kube-system
resourceVersion: "50203"
selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler
uid: 39821272-40a1-4b3a-95bd-a4f09af09231
root@master:/opt/k8s/work# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://192.168.0.107:10259/healthz
ok
root@master:/opt/k8s/work# curl http://192.168.0.107:10251/healthz
ok
部署worker节点（在master节点上执行）
配置kubelet
kubelet 运行在每个 worker 节点上，接收 kube-apiserver 发送的请求，管理 Pod 容器，执行交互式命令，如 exec、run、logs 等。
kubelet 启动时自动向 kube-apiserver 注册节点信息，内置的 cadvisor 统计和监控节点的资源使用情况。
为确保安全，部署时关闭了 kubelet 的非安全 http 端口，对请求进行认证和授权，拒绝未授权的访问(如 apiserver、heapster 的请求)。
创建 kubelet bootstrap kubeconfig 文件
cd /opt/k8s/work
export KUBE_APISERVER=https://192.168.0.107:6443
export node_name=slave
export BOOTSTRAP_TOKEN=$(kubeadm token create \
--description kubelet-bootstrap-token \
--groups system:bootstrappers:${node_name} \
--kubeconfig ~/.kube/config)
# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/cert/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kubelet-bootstrap.kubeconfig
# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=kubelet-bootstrap.kubeconfig
# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=kubelet-bootstrap.kubeconfig
# 设置默认上下文
kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig
向 kubeconfig 写入的是 token，bootstrap 结束后 kube-controller-manager 为 kubelet 创建 client 和 server 证书
kube-apiserver 接收 kubelet 的 bootstrap token 后，将请求的 user 设置为 system:bootstrap:，group 设置为 system:bootstrappers，后续将为这个 group 设置 ClusterRoleBinding
分发 bootstrap kubeconfig 文件到所有 worker 节点
cd /opt/k8s/work
export node_ip=192.168.0.114
scp kubelet-bootstrap.kubeconfig root@${node_ip}:/etc/kubernetes/kubelet-bootstrap.kubeconfig
创建和分发 kubelet 参数配置文件
从 v1.10 开始，部分 kubelet 参数需在配置文件中配置，kubelet --help 会提示
cd /opt/k8s/work
export CLUSTER_CIDR="172.30.0.0/16"
export NODE_IP=192.168.0.114
export CLUSTER_DNS_SVC_IP="10.254.0.2"
cat > kubelet-config.yaml <<EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: ${NODE_IP}
staticPodPath: "/etc/kubernetes/manifests"
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
authentication:
anonymous:
enabled: false
webhook:
enabled: true
x509:
clientCAFile: "/etc/kubernetes/cert/ca.pem"
authorization:
mode: Webhook
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: ${NODE_IP}
clusterDomain: "cluster.local"
clusterDNS:
- "${CLUSTER_DNS_SVC_IP}"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: cgroupfs
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "${CLUSTER_CIDR}"
podPidsLimit: -1
resolvConf: /run/systemd/resolve/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:
memory.available: "100Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]
EOF
address：kubelet 安全端口（https，10250）监听的地址，不能为 127.0.0.1，否则 kube-apiserver、heapster 等不能调用 kubelet 的 API；
readOnlyPort=0：关闭只读端口(默认 10255)，等效为未指定；
authentication.anonymous.enabled：设置为 false，不允许匿名访问 10250 端口；
authentication.x509.clientCAFile：指定签名客户端证书的 CA 证书，开启 HTTP 证书认证；
authentication.webhook.enabled=true：开启 HTTPs bearer token 认证；
对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端)，将被拒绝，提示 Unauthorized；
authroization.mode=Webhook：kubelet 使用 SubjectAccessReview API 查询 kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC)；
featureGates.RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate：自动 rotate 证书，证书的有效期取决于 kube-controller-manager 的 --experimental-cluster-signing-duration 参数
为各节点创建和分发 kubelet 配置文件
cd /opt/k8s/work
export node_ip=192.168.0.114
scp kubelet-config.yaml root@${node_ip}:/etc/kubernetes/kubelet-config.yaml
创建和分发 kubelet 服务启动文件
cd /opt/k8s/work
export K8S_DIR=/data/k8s/k8s
export NODE_NAME=slave
cat > kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=${K8S_DIR}/kubelet
ExecStart=/opt/k8s/bin/kubelet \\
--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\
--cert-dir=/etc/kubernetes/cert \\
--root-dir=${K8S_DIR}/kubelet \\
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
--config=/etc/kubernetes/kubelet-config.yaml \\
--hostname-override=${NODE_NAME} \\
--image-pull-progress-deadline=15m \\
--volume-plugin-dir=${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/ \\
--logtostderr=true \\
--v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF
如果设置了 --hostname-override 选项，则 kube-proxy 也需要设置该选项，否则会出现找不到 Node 的情况；
--bootstrap-kubeconfig：指向 bootstrap kubeconfig 文件，kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求；
K8S approve kubelet 的 csr 请求后，在 --cert-dir 目录创建证书和私钥文件，然后写入 --kubeconfig 文件
安装分发kubelet服务文件
cd /opt/k8s/work
export node_ip=192.168.0.114
scp kubelet.service root@${node_ip}:/etc/systemd/system/kubelet.service
授予 kube-apiserver 访问 kubelet API 的权限
在执行 kubectl exec、run、logs 等命令时，apiserver 会将请求转发到 kubelet 的 https 端口。这里定义 RBAC 规则，授权 apiserver 使用的证书（kubernetes.pem）用户名（CN：kubernetes-api）访问 kubelet API 的权限：
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes-api
Bootstrap Token Auth 和授予权限
kubelet 启动时查找 --kubeletconfig 参数对应的文件是否存在，如果不存在则使用 --bootstrap-kubeconfig 指定的 kubeconfig 文件向 kube-apiserver 发送证书签名请求 (CSR)。
kube-apiserver 收到 CSR 请求后，对其中的 Token 进行认证，认证通过后将请求的 user 设置为 system:bootstrap:，group 设置为 system:bootstrappers，这一过程称为 Bootstrap Token Auth。
默认情况下，这个 user 和 group 没有创建 CSR 的权限, 需要创建一个 clusterrolebinding，将 group system:bootstrappers 和 clusterrole system:node-bootstrapper 绑定：
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
启动 kubelet 服务
export K8S_DIR=/data/k8s/k8s
export node_ip=192.168.0.114
ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/"
ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"
kubelet 启动后使用 --bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求，当这个 CSR 被 approve 后，kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥和 --kubeletconfig 文件。
注意：kube-controller-manager 需要配置 --cluster-signing-cert-file 和 --cluster-signing-key-file 参数，才会为 TLS Bootstrap 创建证书和私钥。
遇到问题
启动kubelet后，使用 kubectl get csr 没有结果，查看kubelet出现错误
journalctl -u kubelet -a |grep -A 2 'certificate_manager.go'
Failed while requesting a signed certificate from the master: cannot create certificate signing request: Unauthorized
查看kube-api服务日志
root@master:/opt/k8s/work# journalctl -eu kube-apiserver
Unable to authenticate the request due to an error: invalid bearer token
原因，在kube-apiserver服务的启动文件中丢掉了下面的配置
--enable-bootstrap-token-auth \\
追加上，重新启动kube-apiserver后解决
kubelet 启动后持续不断的产生csr，手动approve后还继续产生
原因是kube-controller-manager服务停止掉了，重新启动后解决
kubelet服务出问题后 要删除对应节点的/etc/kubernetes/kubelet.kubeconfig和/etc/kubernetes/cert/kubelet-client-current*.pem、/etc/kubernetes/cert/kubelet-client-current*.pem,之后再重新启动kubelet
查看 kubelet 情况
root@master:/opt/k8s/work# kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-kl5mg 49s system:bootstrap:5t989l Pending
csr-mrmkf 2m1s system:bootstrap:5t989l Pending
csr-ql68g 13s system:bootstrap:5t989l Pending
csr-rvl2v 84s system:bootstrap:5t989l Pending
执行时，在手动approve之前会一直追加csr
手动 approve csr
root@master:/opt/k8s/work# kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
certificatesigningrequest.certificates.k8s.io/csr-kl5mg approved
certificatesigningrequest.certificates.k8s.io/csr-mrmkf approved
certificatesigningrequest.certificates.k8s.io/csr-ql68g approved
certificatesigningrequest.certificates.k8s.io/csr-rvl2v approved
root@master:/opt/k8s/work# kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
certificatesigningrequest.certificates.k8s.io/csr-f4smx approved
查看node信息
root@master:/opt/k8s/work# kubectl get nodes
NAME STATUS ROLES AGE VERSION
slave Ready <none> 10m v1.17.2
查看kubelet服务状态
export node_ip=192.168.0.114
root@master:/opt/k8s/work# ssh root@${node_ip} "systemctl status kubelet.service"
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-02-10 22:48:41 CST; 12min ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 15529 (kubelet)
Tasks: 19 (limit: 4541)
CGroup: /system.slice/kubelet.service
└─15529 /opt/k8s/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig --cert-dir=/etc/kubernetes/cert --root-dir=/data/k8s/k8s/kubelet --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/kubelet-config.yaml --hostname-override=slave --image-pull-progress-deadline=15m --volume-plugin-dir=/data/k8s/k8s/kubelet/kubelet-plugins/volume/exec/ --logtostderr=true --v=2
2月 10 22:49:04 slave kubelet[15529]: I0210 22:49:04.846285 15529 kubelet_node_status.go:73] Successfully registered node slave
2月 10 22:49:04 slave kubelet[15529]: I0210 22:49:04.930745 15529 certificate_manager.go:402] Rotating certificates
2月 10 22:49:14 slave kubelet[15529]: I0210 22:49:14.966351 15529 kubelet_node_status.go:486] Recording NodeReady event message for node slave
2月 10 22:49:29 slave kubelet[15529]: I0210 22:49:29.580410 15529 certificate_manager.go:531] Certificate expiration is 2030-02-06 04:19:00 +0000 UTC, rotation deadline is 2029-01-21 13:08:18.850930128 +0000 UTC
2月 10 22:49:29 slave kubelet[15529]: I0210 22:49:29.580484 15529 certificate_manager.go:281] Waiting 78430h18m49.270459727s for next certificate rotation
2月 10 22:49:30 slave kubelet[15529]: I0210 22:49:30.580981 15529 certificate_manager.go:531] Certificate expiration is 2030-02-06 04:19:00 +0000 UTC, rotation deadline is 2027-07-14 16:09:26.990162158 +0000 UTC
2月 10 22:49:30 slave kubelet[15529]: I0210 22:49:30.581096 15529 certificate_manager.go:281] Waiting 65065h19m56.409078053s for next certificate rotation
2月 10 22:53:44 slave kubelet[15529]: I0210 22:53:44.911705 15529 kubelet.go:1312] Image garbage collection succeeded
2月 10 22:53:45 slave kubelet[15529]: I0210 22:53:45.053792 15529 container_manager_linux.go:469] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
2月 10 22:58:45 slave kubelet[15529]: I0210 22:58:45.054225 15529 container_manager_linux.go:469] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.servic
配置kube-proxy 组件
创建 kube-proxy 证书和私钥
创建证书签名请求文件
cd /opt/k8s/work
cat > kube-proxy-csr.json <<EOF
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "NanJing",
"L": "NanJing",
"O": "system:kube-proxy",
"OU": "system"
}
]
}
EOF
CN：指定该证书的 User 为 system:kube-proxy；
预定义的 RoleBinding system:node-proxier 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限。
生成证书和私钥
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
ls kube-proxy*pem
安装证书
cd /opt/k8s/work
export node_ip=192.168.0.114
scp kube-proxy*.pem root@${node_ip}:/etc/kubernetes/cert/
创建 kubeconfig 文件
kube-proxy 使用此文件访问apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-proxy证书等信息
cd /opt/k8s/work
export KUBE_APISERVER=https://192.168.0.107:6443
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/k8s/work/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \
--client-certificate=kube-proxy.pem \
--client-key=kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
分发 kubeconfig
cd /opt/k8s/work
export node_ip=192.168.0.114
scp kube-proxy.kubeconfig root@${node_ip}:/etc/kubernetes/kube-proxy.kubeconfig
创建 kube-proxy 配置文件
cd /opt/k8s/work
export CLUSTER_CIDR="172.30.0.0/16"
export NODE_IP=192.168.0.114
export NODE_NAME=slave
cat > kube-proxy-config.yaml <<EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
burst: 200
kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"
qps: 100
bindAddress: ${NODE_IP}
healthzBindAddress: ${NODE_IP}:10256
metricsBindAddress: ${NODE_IP}:10249
enableProfiling: true
clusterCIDR: ${CLUSTER_CIDR}
hostnameOverride: ${NODE_NAME}
mode: "ipvs"
portRange: ""
iptables:
masqueradeAll: false
ipvs:
scheduler: rr
excludeCIDRs: []
EOF
bindAddress: 监听地址；
clientConnection.kubeconfig: 连接 apiserver 的 kubeconfig 文件；
clusterCIDR: kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量，指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；
hostnameOverride: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；
mode: 使用 ipvs 模式；
分发kube-proxy 配置文件
cd /opt/k8s/work
export node_ip=192.168.0.114
scp kube-proxy-config.yaml root@${node_ip}:/etc/kubernetes/kube-proxy-config.yaml
创建kube-proxy服务启动文件
cd /opt/k8s/work
export K8S_DIR=/data/k8s/k8s
cat > kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=${K8S_DIR}/kube-proxy
ExecStart=/opt/k8s/bin/kube-proxy \\
--config=/etc/kubernetes/kube-proxy-config.yaml \\
--logtostderr=true \\
--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
分发 kube-proxy服务启动文件：
export node_ip=192.168.0.114
scp kube-proxy.service root@${node_ip}:/etc/systemd/system/
启动 kube-proxy服务
export node_ip=192.168.0.114
export K8S_DIR=/data/k8s/k8s
ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-proxy"
ssh root@${node_ip} "modprobe ip_vs_rr"
ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy"
检查启动结果
export node_ip=192.168.0.114
ssh root@${node_ip} "systemctl status kube-proxy |grep Active"
确保状态为 active (running)，否则查看日志，确认原因
如果出现异常，通过如下命令查看
journalctl -u kube-proxy
查看状态
root@slave:~# netstat -lnpt|grep kube-prox
tcp 0 0 192.168.0.114:10256 0.0.0.0:* LISTEN 23078/kube-proxy
tcp 0 0 192.168.0.114:10249 0.0.0.0:* LISTEN 23078/kube-proxy
root@slave:~# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.254.0.1:443 rr
-> 192.168.0.107:6443 Masq 1 0 0
验证集群功能(在master节点上执行）
以一个nginx的service和deployment来验证集群功能
创建启动文件
mkdir /opt/k8s/yml
cd /opt/k8s/yml
cat > nginx.yml << EOF
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
type: NodePort
selector:
app: nginx
ports:
- name: http
port: 80
targetPort: 80
nodePort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.9.1
ports:
- containerPort: 80
EOF
启动服务
kubectl create -f nginx.yml
第一次启动时需要下载k8s.gcr.io/pause:3.1镜像，国内无法直接下载，造成服务无法启动，通过下面操作来解决
docker pull kubeimage/pause:3.1
docker tag kubeimage/pause:3.1 k8s.gcr.io/pause:3.1
观察服务启动情况
root@master:/opt/k8s/yml# kubectl get service -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 41h <none>
nginx NodePort 10.254.8.25 <none> 80:8080/TCP 30m app=nginx
root@master:/opt/k8s/yml# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-56f8998dbc-955gf 1/1 Running 0 30m 172.30.78.2 slave <none> <none>
root@master:/opt/k8s/yml# curl http://192.168.0.114:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
部署 coredns 插件(在master节点上执行）
下载和配置 coredns
cd /opt/k8s/work
git clone https://github.com/coredns/deployment.git
mv deployment coredns
启动 coredns
cd /opt/k8s/work/coredns/kubernetes
export CLUSTER_DNS_SVC_IP="10.254.0.2"
export CLUSTER_DNS_DOMAIN="cluster.local"
./deploy.sh -i ${CLUSTER_DNS_SVC_IP} -d ${CLUSTER_DNS_DOMAIN} | kubectl apply -f -
遇到问题
启动coredns后，状态是CrashLoopBackOff
root@master:/opt/k8s/work/coredns/kubernetes# kubectl get pod -n kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-76b74f549-99bxd 0/1 CrashLoopBackOff 5 4m45s
```
查看coredns对应的pod日志有如下错误
```
root@master:/opt/k8s/work/coredns/kubernetes# kubectl -n kube-system logs coredns-76b74f549-99bxd
.:53
[INFO] plugin/reload: Running configuration MD5 = 8b19e11d5b2a72fb8e63383b064116a1
CoreDNS-1.6.6
linux/amd64, go1.13.5, 6a7a75e
[FATAL] plugin/loop: Loop (127.0.0.1:60429 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 6292641803451309721.7599235642583168995."
```
按照提示进入https://coredns.io/plugins/loop#troubleshooting页面,有如下表述
> When a CoreDNS Pod deployed in Kubernetes detects a loop, the CoreDNS Pod will start to “CrashLoopBackOff”. This is because Kubernetes will try to restart the Pod every time CoreDNS detects the loop and exits.
> A common cause of forwarding loops in Kubernetes clusters is an interaction with a local DNS cache on the host node (e.g. systemd-resolved). For example, in certain configurations systemd-resolved will put the loopback address 127.0.0.53 as a nameserver into /etc/resolv.conf. Kubernetes (via kubelet) by default will pass this /etc/resolv.conf file to all Pods using the default dnsPolicy rendering them unable to make DNS lookups (this includes CoreDNS Pods). CoreDNS uses this /etc/resolv.conf as a list of upstreams to forward requests to. Since it contains a loopback address, CoreDNS ends up forwarding requests to itself.
> There are many ways to work around this issue, some are listed here:
> * Add the following to your kubelet config yaml: resolvConf: <path-to-your-real-resolv-conf-file> (or via command line flag --resolv-conf deprecated in 1.10). Your “real” resolv.conf is the one that contains the actual IPs of your upstream servers, and no local/loopback address. This flag tells kubelet to pass an alternate resolv.conf to Pods. For systems using systemd-resolved, /run/systemd/resolve/resolv.conf is typically the location of the “real” resolv.conf, although this can be different depending on your distribution.
> * Disable the local DNS cache on host nodes, and restore /etc/resolv.conf to the original.
> * A quick and dirty fix is to edit your Corefile, replacing forward . /etc/resolv.conf with the IP address of your upstream DNS, for example forward . 8.8.8.8. But this only fixes the issue for CoreDNS, kubelet will continue to forward the invalid resolv.conf to all default dnsPolicy Pods, leaving them unable to resolve DNS.
按照提示的第一种解决方法，修改kubelet对应的配置文件kubelet-config.yaml中resolv-conf的值为/run/systemd/resolve/resolv.conf,配置片段如下
```
...
podPidsLimit: -1
resolvConf: /run/systemd/resolve/resolv.conf
maxOpenFiles: 1000000
...
```
重启kubelet服务
```
systemctl daemon-reload
systemctl restart kubelet
```
之后重新部署coredns
```
root@master:/opt/k8s/work/coredns/kubernetes# ./deploy.sh -i ${CLUSTER_DNS_SVC_IP} -d ${CLUSTER_DNS_DOMAIN} | kubectl apply -f -
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
root@master:/opt/k8s/work/coredns/kubernetes# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-76b74f549-j5t9c 1/1 Running 0 12s
root@master:/opt/k8s/work/coredns/kubernetes# kubectl get all -n kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
pod/coredns-76b74f549-j5t9c 1/1 Running 0 2m8s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.254.0.2 <none> 53/UDP,53/TCP,9153/TCP 2m8s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 1/1 1 1 2m8s
NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-76b74f549 1 1 1 2m8s
```
启动一个busybox pod，并启动上一章节中验证集群功能的nginx服务，在busybox通过服务名，访问nginx服务
cd /opt/k8s/yml
cat > busybox.yml << EOF
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
- "3600"
EOF
kubectl create -f busybox.yml
kubectl create -f nginx.yml
进入busybox pod中访问nginx
root@master:/opt/k8s/yml# kubectl exec -it busybox sh
/ # cat /etc/resolv.conf
nameserver 10.254.0.2
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ # nslookup www.baidu.com
Server: 10.254.0.2
Address: 10.254.0.2:53
Non-authoritative answer:
www.baidu.com canonical name = www.a.shifen.com
Name: www.a.shifen.com
Address: 183.232.231.174
Name: www.a.shifen.com
Address: 183.232.231.172
/ # nslookup kubernetes
Server: 10.254.0.2
Address: 10.254.0.2:53
Name: kubernetes.default.svc.cluster.local
Address: 10.254.0.1
/ # nslookup nginx
Server: 10.254.0.2
Address: 10.254.0.2:53
Name: nginx.default.svc.cluster.local
Address: 10.254.19.32
/ # ping -c 1 nginx
PING nginx (10.254.19.32): 56 data bytes
64 bytes from 10.254.19.32: seq=0 ttl=64 time=0.155 ms
--- nginx ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.155/0.155/0.155 ms
追加节点（在master上执行）
追加节点
资源有限，我们这边尝试把master节点追加到集群中，如果是新机器，需要执行本文档的 安装前准备，把ca相关的证书分发到这个机器上，部署 flannel 网络步骤
安装前准备（master节点已做过）
把ca相关的证书分发到这个机器上（master节点已做过）
部署 flannel 网络（master节点已做过）
安装docker服务
安装kubelet服务
参照之前追加salve节点的操作，如果直接使用之前的kubelet-bootstrap.yml，发现节点无法加入，因为kubelet-bootstrap.yml中的token值有效期只有一天，如果token已经过期，在kube-apiserver中会出现错误
2月 12 11:01:01 master kube-apiserver[5018]: E0212 11:01:01.640497 5018 authentication.go:104] Unable to authenticate the request due to an error: invalid bearer token
查看token
root@master:/opt/k8s/work# kubeadm token list --kubeconfig ~/.kube/config
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
5t989l.rweut7kedj7ifl1a <invalid> 2020-02-11T18:19:41+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:slave
此时需要按照slave节点上安装kubelet的步骤，重新生成kubelet-bootstrap.yml
将csr approve后，查看节点情况
root@master:/opt/k8s/work# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready <none> 21s v1.17.2
slave Ready <none> 36h v1.17.2
安装kubeproxy服务
重新验证集群
root@master:/opt/k8s/yml# kubectl create -f nginx.yml
service/nginx created
deployment.apps/nginx-deployment created
root@master:/opt/k8s/yml# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-56f8998dbc-6b6qm 1/1 Running 0 87s 172.30.22.2 master <none> <none>
root@master:/opt/k8s/yml# kubectl create -f busybox.yml
pod/busybox created
root@master:/opt/k8s/yml# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 102s 172.30.22.3 master <none> <none>
nginx-deployment-56f8998dbc-6b6qm 1/1 Running 0 3m20s 172.30.22.2 master <none> <none>
root@master:/opt/k8s/yml# curl http://192.168.0.107:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@master:/opt/k8s/yml# curl http://192.168.0.114:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
可以看到访问集群中任意一个节点的8080端口，都可以正确访问到后端对应的nginx服务
![PIP升级+安装Django命令[WARNING: Retrying (Retry(total=4, connect=None...]](https://www.kunjuke.com/file/css/thumb/3.jpg/280-180.jpg)
