kubeadm 部署企业高可用集群

原创

旺旺仙贝哟 2022-10-24 10:29:56 博主文章分类：kubernets ©著作权

文章标签 docker linux sed 文章分类 kubernetes 云计算 在51CTO的第一篇博文

©著作权归作者所有：来自51CTO博客作者旺旺仙贝哟的原创作品，请联系作者获取转载授权，否则将追究法律责任

kubeadm 部署高可用k8s

一，环境初始化

系统规划

主机节点hostname	ip
k8s-master01	192.168.100.129
k8s-master02	192.168.100.138
k8s-master03	192.168.100.139
k8s-node01	192.168.100.140
k8s-node02	192.168.100.141
k8s-node03	192.168.100.142
vip	192.168.100.150

所有节点修改/etc/hosts配置文件 && 修改主机名

[root@k8s-master01~]# hostnamectl set-hostname 主机名
[root@k8s-master01~]#cat >> /etc/hosts << EOF
192.168.100.129 k8s-master01
192.168.100.138 k8s-master02
192.168.100.139 k8s-master03  
192.168.100.140 k8s-node01  gfs1
192.168.100.141 k8s-node02  gfs2
192.168.100.142 k8s-node03  gfs3
EOF

创建秘钥

在k8s-master01执行
ssh-keygen -t rsa
一路回车即可
#将秘钥拷贝到其他主机
[root@k8s-master01~]#for n in { 2 3 };do ssh-copy-id k8s-master0$n;done
[root@k8s-master01~]#for n in { 1 2 3 };do ssh-copy-id k8s-node0$n;done

所有节点安装节点必备工具

[root@k8s-master01~]#yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git -y

修改所有节点yum源

[root@k8s-master01~]#curl -o /etc/yum.repos.d/CentOS-Base.repo
https://mirrors.aliyun.com/repo/Centos-7.repo

[root@k8s-master01~]#yum install -y yum-utils device-mapper-persistent-data lvm2

[root@k8s-master01~]#cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

[root@k8s-master01~]#sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

关闭所有节点防火墙、selinux、dnsmasq、swap

systemctl disable --now firewalld
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config 
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux 
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

所有节点安装ntpdate

[root@k8s-master01~]#yum install ntpdate -y

所有节点配置时间同步（可以使用自己的时间服务器）

ln -sf /usr/share/zoneinfo/Asia/Shanghai  /etc/localtime 
echo 'Asia/Shanghai' > /etc/timezone
crontab -e
*/5 * * * * ntpdate time2.aliyun.com

所有节点配置limit

[root@k8s-master01 ~]# echo "ulimit 65535" >> /etx/profile
ulimit 65535 
[root@k8s-master01 ~]# source /etc/profie

二，内核配置

所有节点升级内核

#2.1：导入ELRepo公钥
[root@k8s-master01 ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
#2.2：安装ELRepo
[root@k8s-master01 ~]# yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
#2.3:安装kernel-it内核
[root@k8s-master01 ~]# yum -y --disablerepo='*' --enablerepo=elrepo-kernel install kernel-lt
#2.4：查看启动器
[root@k8s-master01 ~]# awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
#2.5：设置默认内核启动项 然后重启服务器
[root@k8s-master01 ~]# grub2-set-default 0

所有节点安装ipvsadm

[root@k8s-master01~]# yum install ipvsadm ipset sysstat conntrack libseccomp -y --nogpgcheck

所有节点配置ipvs模块

[root@k8s-master01~]# vim /etc/modules-load.d/ipvs.conf 
# 加入以下内容
cat <<EOF > /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

加载内核配置

[root@k8s-master01~]# systemctl enable --now systemd-modules-load

开启一些k8s集群中必须的内核参数，所有节点配置k8s内核

[root@k8s-master01 ~]# cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
[root@k8s-master01 ~]# sysctl --system

所有节点重启

[root@k8s-master01 ~]# reboot
#重启后查看内核版本
[root@k8s-master01 ~]# uname -r

三，基本组件安装

所有节点安装Docker-ce 19.03

# docker yum源
[root@k8s-master01 ~]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# 安装Docker-ce 19.03
[root@k8s-master01 ~]# yum install docker-ce-19.03.* -y

新版kubelet建议使用systemed，将所有节点docker的CgroupDriver改成systemd

[root@k8s-master01 ~]# mkdir /etc/docker
[root@k8s-master01 ~]# cat > /etc/docker/daemon.json <<EOF
{
 "exec-opts":["native.cgroupdriver=systemd"]
}
EOF

所有节点设置开机自启动Docker

[root@k8s-master01 ~]# systemctl daemon-reload && systemctl enable --now docker
[root@k8s-master01 ~]# systemctl status docker   #查看启动状态

安装k8s组件

所有节点安装置顶版本kubeadm kubelet

[root@k8s-master02 yaml]# yum -y install  kubelet-1.22.8-0.x86_64  #下载指定的版本
[root@k8s-master02 yaml]# yum -y install kubeadm-1.22.8-0.x86_64

默认配置的pause镜像使用gcr.io仓库，国内可能无法访问，所以这里配置Kubelet使用阿里云的pause镜像：

[root@k8s-master01 ~]# cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
EOF

设置Kubelet开机自启动 ( 如果启动失败无须管理，初始化成功以后即可启动 )

[root@k8s-master01 ~]# systemctl daemon-reload
[root@k8s-master01 ~]# systemctl enable --now kubelet

四，高可用组件安装

所有Master节点通过yum安装HAProxy和KeepAlived：

[root@k8s-master01 ~]# yum install keepalived haproxy -y

所有Master节点配置HAProxy（详细配置参考HAProxy文档，所有Master节点的HAProxy配置相同）：

[root@k8s-master01 ~]# vim /etc/haproxy/haproxy.cfg 
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01   192.168.100.129:6443  check
  server k8s-master02   192.168.100.138:6443  check
  server k8s-master03   192.168.100.139:6443  check

所有Master节点配置KeepAlived，配置不一样，注意区分注意每个节点的IP和网卡（interface参数） Master01节点的配置：

[root@k8s-master01 ~]# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens32
    mcast_src_ip 192.168.100.129
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.100.150
    }
    track_script {
       chk_apiserver
    }
}

Master02节点的配置

[root@k8s-master02 ~]# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens32
    mcast_src_ip 192.168.100.138
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.100.150
    }
    track_script {
       chk_apiserver
    }
}

Master03节点的配置

[root@k8s-master03 ~]# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens32
    mcast_src_ip 192.168.100.139
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.100.150
    }
    track_script {
       chk_apiserver
    }
}

所有Master节点配置KeepAlived健康检查文件：

#!/bin/bash
err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

[root@k8s-master01 ~]# chmod +x /etc/keepalived/check_apiserver.sh
[root@k8s-master01 ~]# systemctl daemon-reload
[root@k8s-master01 ~]# systemctl enable --now haproxy
[root@k8s-master01 ~]# systemctl enable --now keepalived

测试VIP

[root@k8s-master01 ~]# ping -c2 -w2 192.168.100.150
PING 192.168.100.150 (192.168.100.150) 56(84) bytes of data.
64 bytes from 192.168.100.150: icmp_seq=1 ttl=64 time=0.336 ms
64 bytes from 192.168.100.150: icmp_seq=2 ttl=64 time=0.222 ms

--- 192.168.100.150 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1013ms
rtt min/avg/max/mdev = 0.222/0.279/0.336/0.057 ms

五，更换集群默认证书

默认情况下，kubeadm 会生成一个集群所需的全部证书。而由于 kubeadm 生成的 CA 根证书有效期为 10 年，客户端证书有效期只有 1 年，因此在一年之内就需要对所有的客户端证书进行续签。所有证书都存放在 /etc/kubernetes/pki 目录下

1.0查看证书过期时间

[root@k8s-master01 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jul 10, 2122 03:09 UTC   344d             ca                      no      
apiserver                  Jul 10, 2122 03:09 UTC   344d             ca                      no      
apiserver-etcd-client      Jul 10, 2122 03:09 UTC   344d             etcd-ca                 no      
apiserver-kubelet-client   Jul 10, 2122 03:09 UTC   344d             ca                      no      
controller-manager.conf    Jul 10, 2122 03:09 UTC   344d             ca                      no      
etcd-healthcheck-client    Jul 10, 2122 03:09 UTC   344d             etcd-ca                 no      
etcd-peer                  Jul 10, 2122 03:09 UTC   344d             etcd-ca                 no      
etcd-server                Jul 10, 2122 03:09 UTC   344d             etcd-ca                 no      
front-proxy-client         Jul 10, 2122 03:09 UTC   344d             front-proxy-ca          no      
scheduler.conf             Jul 10, 2122 03:09 UTC   344d             ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jul 29, 2032 10:13 UTC   9y              no      
etcd-ca                 Jul 29, 2032 10:13 UTC   9y              no      
front-proxy-ca          Jul 29, 2032 10:13 UTC   9y              no

2.0 集群根证书

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/ca*
-rw-r--r-- 1 root root 1066 Aug  7 03:03 /etc/kubernetes/pki/ca.crt
-rw------- 1 root root 1675 Aug  7 03:03 /etc/kubernetes/pki/ca.key

由此集群根证书签发的证书有：

kube-apiserver 组件持有的服务端证书

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/apiserver.*
-rw-r--r-- 1 root root 1277 Aug  7 03:03 /etc/kubernetes/pki/apiserver.crt
-rw------- 1 root root 1675 Aug  7 03:03 /etc/kubernetes/pki/apiserver.key

kubelet 组件持有的客户端证书

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/apiserver-kubelet-client.*
-rw-r--r-- 1 root root 1143 Aug  7 03:03 /etc/kubernetes/pki/apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Aug  7 03:03 /etc/kubernetes/pki/apiserver-kubelet-client.key

说明：kubelet 的 /var/lib/kubelet/config.yaml 配置文件中一般不会明确指定服务端证书，而是只指定 ca 根证书，让 kubelet 根据本地主机信息自动生成服务端证书并保存到配置的 cert-dir 文件夹中。

3.0 汇聚层根证书

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/front-proxy-ca.*
-rw-r--r-- 1 root root 1078 Aug  7 03:03 /etc/kubernetes/pki/front-proxy-ca.crt
-rw------- 1 root root 1675 Aug  7 03:03 /etc/kubernetes/pki/front-proxy-ca.key

由此汇聚层根证书签发的证书有：

代理端使用的客户端证书，用作代用户与 kube-apiserver 认证

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/front-proxy-client.*
-rw-r--r-- 1 root root 1103 Aug  7 03:03 /etc/kubernetes/pki/front-proxy-client.crt
-rw------- 1 root root 1675 Aug  7 03:03 /etc/kubernetes/pki/front-proxy-client.key

4.0 etcd集群根证书

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/etcd/ca.*
-rw-r--r-- 1 root root 1058 Aug  7 03:03 /etc/kubernetes/pki/etcd/ca.crt
-rw------- 1 root root 1679 Aug  7 03:03 /etc/kubernetes/pki/etcd/ca.key

由此 etcd 根证书签发的证书有：

etcd server 服务端证书

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/etcd/server.*
-rw-r--r-- 1 root root 1188 Aug  7 03:03 /etc/kubernetes/pki/etcd/server.crt
-rw------- 1 root root 1679 Aug  7 03:03 /etc/kubernetes/pki/etcd/server.key

Pod 中定义 Liveness 探针使用的客户端证书

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/etcd/healthcheck-client.*
-rw-r--r-- 1 root root 1139 Aug  7 03:03 /etc/kubernetes/pki/etcd/healthcheck-client.crt
-rw------- 1 root root 1675 Aug  7 03:03 /etc/kubernetes/pki/etcd/healthcheck-client.key

配置在 kube-apiserver 中用来与 etcd server 做双向认证的客户端证书

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/apiserver-etcd-client.*
-rw-r--r-- 1 root root 1135 Aug  7 03:03 /etc/kubernetes/pki/apiserver-etcd-client.crt
-rw------- 1 root root 1679 Aug  7 03:03 /etc/kubernetes/pki/apiserver-etcd-client.key

5.0 Serveice Account 秘钥

[root@k8s-master01 ~]# ll /etc/kubernetes/pki/sa.*
-rw------- 1 root root 1675 Aug  7 03:03 /etc/kubernetes/pki/sa.key
-rw------- 1 root root  451 Aug  7 03:03 /etc/kubernetes/pki/sa.pub

说明：Serveice Account 密钥对仅提供给 kube-controller-manager 使用。kube-controller-manager 通过 sa.key 对 token 进行签名，Master 节点通过公钥 sa.pub 进行签名的验证。
延伸 ——API Server 身份验证过程
API Server 的 Authentication 环节支持多种身份校验方式：Client Cert、Bearer Token、Static Password Auth 等，这些方式中只要有一种方式通过 Authentication（Kubernetes API Server 会逐个方式尝试），那么身份校验就会通过。
一旦 API Server 发现 Client 发起的 Request 使用的是 Service Account Token 的方式，API Server 就会自动采用 Signed Bearer Token 方式进行身份校验。而 Request 就会使用携带的 Service Account Token 参与验证。该 Token 是 API Server 在创建 Service Account 时用 kube-controller-manager 启动参数 --service-account-private-key-file 指定的私钥签署 (sign) 的，同时必须指定 kube-apiserver 参数 --service-account-key-file（如果没指定的话，会使用 --tls-private-key-file 替代）为该私钥对应的公钥，用来在认证阶段验证 Token，也就是说该证书对通过 CN 和 O 指定了 ServiceAccount 的授权权限。
通过 Authentication 后，API Server 将根据 Pod 所属 ServiceAccount 的用户名（以 system:serviceaccount: 为前缀）和组（以 system:serviceaccounts: 前缀）的权限对其进行 Authorization 和 Admission Control 两个环节的处理。
不管是自动生成的 Token 还是手动创建的 Token 的值都是一样的，因为进行签署 Token 的 -–service-account-key-file 是同一个。
ServiceAccount 中的 Token 是 API server 私钥签署的，Pod 在对 API Server 发起请求的时候会带上该 Token，以确保能够通过 API Server 的认证。对 ServiceAccount 的授权通过对 ServiceAccount 对应的用户或组进行 RBAC 控制即可

6.0 查看当前版本

在任意一台 Master 节点执行以下命令查看当前版本，GitVersion 对应 Kubernetes 版本，GoVersion 对应使用的 Golang 版本

[root@master1~]# kubeadm version 
kubeadm version: &version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.8-dirty", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"dirty", BuildDate:"2022-08-03T03:06:52Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

7.0 修改 kubeadm 源码并重新编译

安装 Golang 等编译源码的环境包

安装编译工具

[root@master1~]# yum install -y gcc make rsync jq

下载并配置 Golang 环境

## 下载 golang 1.16.15
[root@master1~]# wget https://dl.google.com/go/go1.16.15.linux-amd64.tar.gz

## 解压并放置在一个目录中
[root@master1~]# tar zxvf go1.16.15.linux-amd64.tar.gz  -C /usr/local

## 编辑 /etc/profile 文件,添加 Go 环境配置内容
[root@master1~]# vi /etc/profile

export GOROOT=/usr/local/go
export GOPATH=/usr/local/gopath
export PATH=$PATH:$GOROOT/bin

## 使配置生效
[root@master1~]# source /etc/profile

## 测试 Go 命令是否配置成功,成功则显示如下
[root@master1~]# go version
go version go1.16.15 linux/amd64

8.0 获取源码

获取源码，以 v1.22.8 为例

[root@master1~]# yum install git wget -y
[root@master1~]# cd /root/
[root@master1~]# git clone --branch v1.22.8 --single-branch --depth 1 https://github.com/kubernetes/kubernetes

or

## 下的 kubernetes 源码
[root@master1~]# git clone https://github.com/kubernetes/kubernetes.git
## 进入 Kubernetes 目录
[root@master1~]# cd kubernetes
## 切换 Kubernetes 版本
[root@master1~]# git checkout v1.22.8

接下来我们修改 Kubernetes 代码中与 kubeadm 证书有效期相关的源码，操作的命令如下:

修改 constants.go 文件，操作如下:

[root@master1~]# vim cmd/kubeadm/app/constants/constants.go


...
const (
        // KubernetesDir is the directory Kubernetes owns for storing various configuration files
        KubernetesDir = "/etc/kubernetes"
        // ManifestsSubDirName defines directory name to store manifests
        ManifestsSubDirName = "manifests"
        // TempDirForKubeadm defines temporary directory for kubeadm
        // should be joined with KubernetesDir.
        TempDirForKubeadm = "tmp"

        // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        // 修改time.Hour * 24 * 365为time.Hour * 24 * 365 * 100
        CertificateValidity = time.Hour * 24 * 365 * 100

        // CACertAndKeyBaseName defines certificate authority base name
        CACertAndKeyBaseName = "ca"
...

修改 cert.go 文件，操作如下:

[root@master1~]# vim staging/src/k8s.io/client-go/util/cert/cert.go


...
// NewSelfSignedCACert creates a CA certificate
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
        now := time.Now()
        tmpl := x509.Certificate{
                SerialNumber: new(big.Int).SetInt64(0),
                Subject: pkix.Name{
                        CommonName:   cfg.CommonName,
                        Organization: cfg.Organization,
                },
                NotBefore:             now.UTC(),
                NotAfter:              now.Add(duration365d * 100).UTC(),  //修改duration365d * 10为duration365d * 100
                KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
                BasicConstraintsValid: true,
                IsCA:                  true,
        }

        certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key)
        if err != nil {
                return nil, err
        }
        return x509.ParseCertificate(certDERBytes)
}
...

9.0 执行 kubeadm 编译

使用 make 命令编译 kubeadm, 执行的命令如下:

[root@master1~]# make all WHAT=cmd/kubeadm GOFLAGS=-v

编译成功后的 kubeadm 会放到当前目录中的 ./_output/local/bin/linux/amd64/ 目录中，我们进入到该文件下，查看是否有对应的文件。

[root@k8s-master01 amd64]# ll
总用量 78648
-rwxr-xr-x 1 root root  5959680 8月  25 09:39 conversion-gen
-rwxr-xr-x 1 root root  5701632 8月  25 09:39 deepcopy-gen
-rwxr-xr-x 1 root root  5722112 8月  25 09:39 defaulter-gen
-rwxr-xr-x 1 root root  3565441 8月  25 09:38 go2make
-rwxr-xr-x 1 root root 45809664 8月  25 09:41 kubeadm
-rwxr-xr-x 1 root root  8105984 8月  25 09:40 openapi-gen
-rwxr-xr-x 1 root root  5668864 8月  25 09:38 prerelease-lifecycle-gen

接下来我们使用自定义编译的 kubeadm 工具，将已经安装的 kubeadm 给替换掉，执行的操作命令如下:

## 备份已有的 kubeadm 工具
[root@master1~]# mv /usr/bin/kubeadm  /usr/bin/kubeadm_backup

## 使用自己编译的 kubeadm 替换现有的 kubeadm 工具
[root@master1~]# cp ./_output/local/bin/linux/amd64/kubeadm  /usr/bin/kubeadm

## 修改文件的执行权限
[root@master1~]# chmod +x /usr/bin/kubeadm

六，初始化集群

Master01节点创建kubeadm-config.yaml配置文件如下：

[root@k8s-master01 ~]# cd /etc/kubernetes/
[root@k8s-master01 ~]# mkdir yaml && cd yaml
[root@k8s-master01 ~]# vim kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 7t2weq.bjbawausm0jaxury
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.100.129
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
  - 192.168.100.150
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.100.150:16443
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.22.8
networking:
  dnsDomain: cluster.local
  podSubnet: 172.168.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

更新kubeadm-config.yaml为new.yaml

[root@k8s-master01 yaml]# kubeadm config migrate --old-config kubeadm-config.yaml --new-config new.yaml

将new.yaml文件复制到其他master节点，之后所有master节点提前下载镜像

#在k8s-master02跟k8s-master03上面创建目录并把k8s-master01上面生成的new.yaml拷贝到yaml目录下面并修改对应的节点名跟ip地址
[root@k8s-master02~]# cd /etc/kubernetes/
[root@k8s-master02~]# mkdir yaml && cd yaml
[root@k8s-master01 yaml]# scp new.yaml master02:/etc/kubernetes/yaml
[root@k8s-master01 yaml]# scp new.yaml master03:/etc/kubernetes/yaml
[root@k8s-master01 yaml]# kubeadm config images pull --config ./new.yaml

Master01节点初始化，初始化以后会在/etc/kubernetes目录下生成对应的证书和配置文件，之后其他Master节点加入Master01即可：

[root@k8s-master01 yaml]# kubeadm init --config ./new.yaml  --upload-certs

如果初始化失败执行如下命令，重新初始化

[root@k8s-master01 yaml]# kubeadm reset -f ; ipvsadm --clear ; rm -rf ~/.kube

初始化成功以后，会产生Token值，用于其他节点加入时使用，因此要记录下初始化成功生成的token值（令牌值）

#此信息需要保存
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.100.150:16443 --token 7t2weq.bjbawausm0jaxury \
  --discovery-token-ca-cert-hash sha256:4074b0b0b56058083407a92201c0f7c4758d8a9f48571a229d8015609092bb93 \
  --control-plane --certificate-key 89213ad9e88494b525248d24038a1571eb751f3ed02ebe5b5b209045b6b7f27a

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.150:16443 --token 7t2weq.bjbawausm0jaxury \
  --discovery-token-ca-cert-hash sha256:4074b0b0b56058083407a92201c0f7c4758d8a9f48571a229d8015609092bb93

如果看到这个输出就证明你已经初始化成功了

Master01节点配置环境变量，用于访问kubernetes集群：

[root@k8s-master01 kubernetes]#mkdir -p $HOME/.kube
[root@k8s-master01 kubernetes]#sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master01 kubernetes]#sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s-master01 kubernetes]# cat <<EOF >> /etc/profile
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
[root@k8s-master01 kubernetes]# source /etc/profile

查看节点状态：

[root@k8s-master01 kubernetes]# kubectl get node
NAME           STATUS     ROLES                  AGE   VERSION
k8s-master01   NotReady   control-plane,master   11m   v1.22.8

七，高可用Master

初始化其他master加入集群,找到刚在k8s-master01上面初始化成功的提示信息

# master 节点加入执行
kubeadm join 192.168.100.150:16443 --token 7t2weq.bjbawausm0jaxury \
  --discovery-token-ca-cert-hash sha256:4074b0b0b56058083407a92201c0f7c4758d8a9f48571a229d8015609092bb93 \
  --control-plane --certificate-key 89213ad9e88494b525248d24038a1571eb751f3ed02ebe5b5b209045b6b7f27a

# node   节点加入执行
kubeadm join 192.168.100.150:16443 --token 7t2weq.bjbawausm0jaxury \
  --discovery-token-ca-cert-hash sha256:4074b0b0b56058083407a92201c0f7c4758d8a9f48571a229d8015609092bb93

Token 过期处理

Token过期后生成新的Token：node

[root@k8s-master01 amd64]# kubeadm token create --print-join-command
kubeadm join 192.168.100.150:16443 --token 6ckadh.lp0j2rosa2720h90 --discovery-token-ca-cert-hash sha256:4074b0b0b56058083407a92201c0f7c4758d8a9f48571a229d8015609092bb93

Master需要生成 --certificate-key

[root@k8s-master01 amd64]# kubeadm init phase upload-certs --upload-certs
I0826 15:30:49.265934   61996 version.go:255] remote version is much newer: v1.25.0; falling back to: stable-1.22
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
d69f6bdd4707f54e01120dfb82e7d4ef40f8fbdbf6fd872e43a751de098e4bda

八，部署网络插件（calico）

安装calico网络插件，用于容器间的通讯

[root@k8s-master01~]# kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

查看容器状态(需要等几分钟)

[root@k8s-master01 kubernetes]# kubectl get pod -A -o wide
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE    IP                NODE           NOMINATED NODE   READINESS GATES
kube-system   coredns-7d89d9b6b8-2pkhk               1/1     Running   0          162m   172.168.0.3       k8s-master01   <none>           <none>
kube-system   coredns-7d89d9b6b8-p5l4v               1/1     Running   0          162m   172.168.0.2       k8s-master01   <none>           <none>
kube-system   etcd-k8s-master01                      1/1     Running   0          163m   192.168.113.100   k8s-master01   <none>           <none>
kube-system   kube-apiserver-k8s-master01            1/1     Running   0          163m   192.168.113.100   k8s-master01   <none>           <none>
kube-system   kube-controller-manager-k8s-master01   1/1     Running   0          163m   192.168.113.100   k8s-master01   <none>           <none>
kube-system   kube-flannel-ds-ghqgt                  1/1     Running   0          2m52s  192.168.113.103   k8s-node01     <none>           <none>
kube-system   kube-flannel-ds-dsada                  1/1     Running   0          3m4s   192.168.113.100   k8s-master01   <none>           <none>
kube-system   kube-proxy-hkssd                       1/1     Running   0          162m   192.168.113.100   k8s-master01   <none>           <none>
kube-system   kube-scheduler-k8s-master01            1/1     Running   0          163m   192.168.113.100   k8s-master01   <none>           <none>

九，Metrics Server部署

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。

将Master01节点的front-proxy-ca.crt复制到所有Node节点

[root@k8s-master01 pki]# for n in { 1 2 3 };do scp  /etc/kubernetes/pki/front-proxy-ca.crt k8s-node0$n:/etc/kubernetes/pki/front-proxy-ca.crt;done

安装metrics server

[root@k8s-master01 ~]# cd /etc/kubernetes/yaml
[root@k8s-master01 yaml~]# git clone https://github.com/dotbalo/k8s-ha-install.git
[root@k8s-master01 yaml~]# cd ./k8s-ha-install && git checkout manual-installation-v1.20.x
[root@k8s-master01 yaml~]# cd ./metrics-server-0.4.x-kubeadm
[root@k8s-master01 yaml~]# kubectl apply -f comp.yaml

查看状态并进行测试

[root@k8s-master01 pki]# kubectl get pod -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS      AGE
kube-system   calico-kube-controllers-58f755f869-s7s62   1/1     Running   0             59m
kube-system   calico-node-bx9gl                          1/1     Running   0             59m
kube-system   calico-node-gt7dd                          1/1     Running   0             59m
kube-system   calico-node-hs5gj                          1/1     Running   0             59m
kube-system   calico-node-mq27r                          1/1     Running   0             59m
kube-system   calico-node-wlw8f                          1/1     Running   0             59m
kube-system   calico-node-zph5q                          1/1     Running   0             59m
kube-system   coredns-7d89d9b6b8-427cx                   1/1     Running   0             16h
kube-system   coredns-7d89d9b6b8-j87gf                   1/1     Running   0             16h
kube-system   etcd-k8s-master01                          1/1     Running   0             16h
kube-system   etcd-master2                               1/1     Running   0             16h
kube-system   etcd-master3                               1/1     Running   0             16h
kube-system   kube-apiserver-k8s-master01                1/1     Running   0             16h
kube-system   kube-apiserver-master2                     1/1     Running   0             16h
kube-system   kube-apiserver-master3                     1/1     Running   1 (16h ago)   16h
kube-system   kube-controller-manager-k8s-master01       1/1     Running   1 (16h ago)   16h
kube-system   kube-controller-manager-master2            1/1     Running   0             16h
kube-system   kube-controller-manager-master3            1/1     Running   0             16h
kube-system   kube-proxy-2c2k7                           1/1     Running   0             16h
kube-system   kube-proxy-8vpk4                           1/1     Running   0             16h
kube-system   kube-proxy-98dxk                           1/1     Running   0             16h
kube-system   kube-proxy-gwxwh                           1/1     Running   0             16h
kube-system   kube-proxy-pm28t                           1/1     Running   0             16h
kube-system   kube-proxy-snn55                           1/1     Running   0             16h
kube-system   kube-scheduler-k8s-master01                1/1     Running   1 (16h ago)   16h
kube-system   kube-scheduler-master2                     1/1     Running   0             16h
kube-system   kube-scheduler-master3                     1/1     Running   0             16h
kube-system   metrics-server-d6c46b546-2pk7x             1/1     Running   0             36s

[root@k8s-master01 amd64]# kubectl top node
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master01   140m         7%     1277Mi          68%       
k8s-master02   121m         6%     1296Mi          69%       
k8s-master03   145m         7%     1259Mi          67%       
k8s-node01     51m          2%     1000Mi          53%       
k8s-node02     50m          2%     1007Mi          54%       
k8s-node03     50m          2%     1013Mi          54%

十，Dashboard 部署

[root@k8s-master01 ~]# cd /etc/kubernetes/yaml/k8s-ha-install/dashboard/    #进入到前面转的资源收集器目录下面
[root@k8s-master01 dashboard]# kubectl apply -f .

更改dashboard的svc为NodePort：

[root@k8s-master01 ~]# kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"k8s-app":"kubernetes-dashboard"},"name":"kubernetes-dashboard","namespace":"kubernetes-dashboard"},"spec":{"ports":[{"port":443,"targetPort":8443}],"selector":{"k8s-app":"kubernetes-dashboard"}}}
  creationTimestamp: "2022-08-02T02:53:24Z"
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  resourceVersion: "115476"
  uid: ac49e74a-addd-4019-87af-5d84d81aaeae
spec:
  clusterIP: 10.105.206.58
  clusterIPs:
  - 10.105.206.58
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

查看令牌
[root@k8s-master01 ~]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

Name:         admin-user-token-szw2c
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: admin-user
              kubernetes.io/service-account.uid: d1d5a524-83c2-40ca-8d29-481e8bb94844

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1099 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IjRsb3lGU0xNem12c0U4Q3NYalRsbUgyOEVfM3VnekRlQlphcU1rcUdnQ28ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXN6dzJjIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkMWQ1YTUyNC04M2MyLTQwY2EtOGQyOS00ODFlOGJiOTQ4NDQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.KCs_rca0jckWGKMQmVEHcAhWG9ttEtYmY1eRZNLCrTdbmp0Ur_mH307DN0GPoAeWemdVYthTx7SkkdT6Q1LUc3EX-998x4vgNnuKlswSD4rcvU6AQc4STuEhwUBcvVa7tA4DbzpAPjUGiqk0QpCUJ4mtLwKVJcg1etdWx1Xo-bekTkRQLkHcDYi2_35Ve2q7Co-Mf7VQFOO7PNz4lzpxCsWxj5aR5PSdNArnDMT6cKX_yX5jhIgMXDJ9Gwp808IKpt9Jo0zB-ij6D_-tL7HoGXNnP0IQSoPYPjARsoAar9hO8GvUhdEpzSWIi3UQU5bbutB0324j7au7tXWqhwqR4g

访问web https://任意节点ip:端口

查看端口

[root@k8s-master01 amd64]# kubectl get svc -A
NAMESPACE              NAME                                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default                glusterfs-dynamic-b7351833-9294-43b2-ba24-f678551df7cc   ClusterIP   10.102.25.120    <none>        1/TCP                    92m
default                kubernetes                                               ClusterIP   10.96.0.1        <none>        443/TCP                  28h
kube-system            kube-dns                                                 ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   28h
kube-system            metrics-server                                           ClusterIP   10.106.51.215    <none>        443/TCP                  25h
kubernetes-dashboard   dashboard-metrics-scraper                                ClusterIP   10.103.187.241   <none>        8000/TCP                 24h
kubernetes-dashboard   kubernetes-dashboard                                     NodePort    10.104.123.13    <none>        443:32490/TCP            24h

查看节点证书

[root@k8s-master01 amd64]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Aug 01, 2122 03:08 UTC   99y             ca                      no      
apiserver                  Aug 01, 2122 03:08 UTC   99y             ca                      no      
apiserver-etcd-client      Aug 01, 2122 03:08 UTC   99y             etcd-ca                 no      
apiserver-kubelet-client   Aug 01, 2122 03:08 UTC   99y             ca                      no      
controller-manager.conf    Aug 01, 2122 03:08 UTC   99y             ca                      no      
etcd-healthcheck-client    Aug 01, 2122 03:08 UTC   99y             etcd-ca                 no      
etcd-peer                  Aug 01, 2122 03:08 UTC   99y             etcd-ca                 no      
etcd-server                Aug 01, 2122 03:08 UTC   99y             etcd-ca                 no      
front-proxy-client         Aug 01, 2122 03:08 UTC   99y             front-proxy-ca          no      
scheduler.conf             Aug 01, 2122 03:08 UTC   99y             ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Aug 01, 2122 03:08 UTC   99y             no      
etcd-ca                 Aug 01, 2122 03:08 UTC   99y             no      
front-proxy-ca          Aug 01, 2122 03:08 UTC   99y             no