一、安装环境准备
1.机器列表
主机名 | IP | 操作系统 | 角色 | 安装软件 |
master | 192.168.0.100 | CentOS 7 | 管理节点 | docker |
node1 | 192.168.0.101 | CentOS 7 | 工作节点 | docker |
node2 | 192.168.0.102 | CentOS 7 | 工作节点 | docker |
2.环境初始化
注意:以下操作在三台机器上都要执行
2.1关闭防火墙及selinux
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/=enforcing/=disabled/g' /etc/selinux/config
2.2关闭swap分区
swapoff -a #临时
sed -i '/swap/s/^/#/' /etc/fstab #永久
默认情况下,kubelet不允许所在的主机存在交换分区,后期规划的时候,可以考虑在系统安装的时候不创建交换分区,针对已经存在交换分区的可以设置忽略禁止使用swap的限制,不然无法启动kubelet。一般直接禁用swap就可以了,不需要执行此步骤。
vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
2.3添加yum仓库
docker-ce仓库
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
k8s仓库
cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
enabled=1
2.4安装docker和kubeadm
默认安装最新版,也可以手动指定版本,如 kubelet-1.20.1
yum install docker-ce kubelet kubeadm kubectl -y
2.5启动docker和kubelet
systemctl start docker && systemctl enable docker
systemctl start kubelet && systemctl enable kubelet
注意,此时kubelet是无法正常启动的,可以查看/var/log/messages有报错信息,等待master节点初始化之后即可正常运行。
2.6提前下载所需镜像
vim k8s-image-download.sh
#!/bin/bash
# download k8s 1.20.1 images
# get image-list by 'kubeadm config images list --kubernetes-version=v1.20.1'
# gcr.azk8s.cn/google-containers == k8s.gcr.io
if [ $# -ne 1 ];then
echo "USAGE: bash `basename $0` KUBERNETES-VERSION"
exit 1
fi
version=$1
images=`kubeadm config images list --kubernetes-version=${version} |awk -F'/' '{print $2}'`
for imageName in ${images[@]};do
docker pull registry.aliyuncs.com/google_containers/$imageName
# docker pull gcr.azk8s.cn/google-containers/$imageName
# docker tag gcr.azk8s.cn/google-containers/$imageName k8s.gcr.io/$imageName
# docker rmi gcr.azk8s.cn/google-containers/$imageName
done
二、集群搭建
1.master节点执行
kubeadm init --kubernetes-version=v1.20.1 \
--pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12 \
--apiserver-advertise-address=192.168.0.10 \
--ignore-preflight-errors=Swap \
--ignore-preflight-errors=NumCPU \
--image-repository registry.aliyuncs.com/google_containers
参数说明
- --kubernetes-version=v1.20.1:指定要安装的版本号。
- --apiserver-advertise-address:指定用 Master 的哪个IP地址与 Cluster的其他节点通信。
- --service-cidr:指定Service网络的范围,即负载均衡VIP使用的IP地址段。
- --pod-network-cidr:指定Pod网络的范围,即Pod的IP地址段。
- --ignore-preflight-errors=:忽略运行时的错误,例如执行时存在[ERROR NumCPU]和[ERROR Swap],忽略这两个报错就是增加--ignore-preflight-errors=NumCPU 和--ignore-preflight-errors=Swap的配置即可。
- --image-repository:Kubenetes默认Registries地址是 k8s.gcr.io,一般在国内并不能访问 gcr.io,可以将其指定为阿里云镜像地址:registry.aliyuncs.com/google_containers。
如果有多个网卡,最好指定一下 apiserver-advertise 地址
执行过程显示如下:
[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
[WARNING NumCPU]: the number of available CPUs 1 is less than the required 2
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [node-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 1.1.1.101]
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [node-1 localhost] and IPs [1.1.1.101 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [node-1 localhost] and IPs [1.1.1.101 127.0.0.1 ::1]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 22.503724 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: z1609x.bg2tkrsrfwlrl3rb
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.100:6443 --token z1609x.bg2tkrsrfwlrl3rb \
--discovery-token-ca-cert-hash sha256:0753a3d2f04c6c34c5ad88d4be3bc508b1e5b9d00908b29442f7068645521703
初始化操作主要经历了下面15个步骤,每个阶段均输出均使用[步骤名称]作为开头:
- [init]:指定版本进行初始化操作
- [preflight] :初始化前的检查和下载所需要的Docker镜像文件。
- [kubelet-start] :生成kubelet的配置文件”/var/lib/kubelet/config.yaml”,没有这个文件kubelet无法启动,所以初始化之前的kubelet实际上启动失败。
- [certificates]:生成Kubernetes使用的证书,存放在/etc/kubernetes/pki目录中。
- [kubeconfig] :生成 KubeConfig 文件,存放在/etc/kubernetes目录中,组件之间通信需要使用对应文件。
- [control-plane]:使用/etc/kubernetes/manifest目录下的YAML文件,安装 Master 组件。
- [etcd]:使用/etc/kubernetes/manifest/etcd.yaml安装Etcd服务。
- [wait-control-plane]:等待control-plan部署的Master组件启动。
- [apiclient]:检查Master组件服务状态。
- [upload-config]:更新配置
- [kubelet]:使用configMap配置kubelet。
- [patchnode]:更新CNI信息到Node上,通过注释的方式记录。
- [mark-control-plane]:为当前节点打标签,打了角色Master,和不可调度标签,这样默认就不会使用Master节点来运行Pod。
- [bootstrap-token]:生成token记录下来,后边使用kubeadm join往集群中添加节点时会用到
- [addons]:安装附加组件CoreDNS和kube-proxy
PS:如果安装失败,可以执行 kubeadm reset 命令将主机恢复原状,重新执行 kubeadm init
kubectl默认会在执行的用户家目录下面的.kube目录下寻找config文件。这里是将在初始化时[kubeconfig]步骤生成的admin.conf拷贝到.kube/config。
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
2.安装网络插件flannel
# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
3.节点加入集群
在各node节点上执行
kubeadm join 192.168.0.100:6443 --token z1609x.bg2tkrsrfwlrl3rb \
--discovery-token-ca-cert-hash sha256:0753a3d2f04c6c34c5ad88d4be3bc508b1e5b9d00908b29442f7068645521703
4.查看集群状态
kubectl get node
kubectl get pod --all-namespaces -o wide
各节点都是 Ready 状态,各Pod都是 Running 状态,表示集群正常运行。
5.测试DNS解析是否正常
kubectl run -it busybox --image=radial/busyboxplus:curl
[ root@busybox:/ ]$ nslookup kubernetes
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
[ root@busybox:/ ]$ nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
[ root@busybox:/ ]$
6.测试集群
在kubernetes集群中创建一个pod,然后暴露端口,验证是否正常访问:
kubectl create deployment nginx-deploy --image=nginx
kubectl expose deployment nginx-deploy --port=80 --type=NodePort
kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/nginx-deploy-8588f9dfb-q9qqd 1/1 Running 0 8h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9h
service/nginx-deploy NodePort 10.104.148.168 <none> 80:31629/TCP 8h
访问地址:http://NodeIP:Port ,此例就是:http://192.168.0.100:31629
说明:
默认token的有效期为24小时,过期之后,该token就不可用了,
如果后续有nodes节点加入,可以重新生成新的token,解决方法如下:
#生成token
kubeadm token create
0w3a92.ijgba9ia0e3scicg
#查看token
kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
0w3a92.ijgba9ia0e3scicg 23h 2019-09-08T22:02:40+08:00 authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
t0ehj8.k4ef3gq0icr3etl0 22h 2019-09-08T20:58:34+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
#获取ca证书sha256编码hash值
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
#节点加入集群
kubeadm join --token aa78f6.8b4cafc8ed26c34f --discovery-token-ca-cert-hash sha256:0fd95a9bc67a7bf0ef42da968a0d55d92e52898ec37c971bd77ee501d845b538 192.168.73.138:6443 --skip-preflight-chec