部署K8S流程:
1、基础环境准备,并关闭防火墙 selinux 和 swap,更新软件源、时间同步、安装常用命、开启路由转发。
2、部署 harbor 及 haproxy keeplivad高可用反向代理
3、在所有 master 安装指定版本的 kubeadm 、kubelet、kubectl、docker
4、在所有 node 节点安装指定版本的 kubeadm 、kubelet、docker,在 node 节点 kubectl 为可选安装,看是否需要在 node 执行 kubectl 命令进行集群管理及 pod 管理等操作。
5、master 节点运行 kubeadm init 初始化命令
6、验证 master 节点状态
7、在 node 节点使用 kubeadm 命令将自己加入 k8s master(需要使用 master 生成 token 认 证)
8、验证 node 节点状态
9、创建 pod 并测试网络通信
10、部署 web 服务 Dashboard
11、k8s 集群升级
192.168.80.100 localhost7A.localdomain harbor CentOS 7.7
192.168.80.110 localhost7B.localdomain keepalived haproxy 192.168.80.222 CentOS 7.7
192.168.80.120 localhost7C.localdomain master 192.168.80.222 CentOS 7.7
192.168.80.130 localhost7D.localdomain master 192.168.80.222 CentOS 7.7
192.168.80.140 localhost7E.localdomain master 192.168.80.222 CentOS 7.7
192.168.80.150 localhost7F.localdomain node1 CentOS 7.7
192.168.80.160 localhost7G.localdomain node2 CentOS 7.7
192.168.80.170 localhost7H.localdomain node3 CentOS 7.7
一、部署 harbor
1.同步时间服务 并关闭防火墙和selinux 开启路由转发
[root@localhost7A ~]# ntpdate time1.aliyun.com && hwclock -w
[root@localhost7A ~]# echo 1 > /proc/sys/net/ipv4/ip_forward
2.下载YUM源
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
3.安装docker docker-compose
yum list docker-ce --showduplicates
yum install docker-ce-19.03.15-3.el7 docker-ce-cli-19.03.15-3.el7 -y
yum install docker-compose -y
systemctl enable docker && systemctl start docker
4.下载harbor 解压
https://github.com/goharbor/harbor/releases/download/v1.7.6/harbor-offline-installer-v1.7.6
tar xvf harbor-offline-installer-v1.7.6.tgz
ln -sv /usr/local/src/harbor /usr/local/
cd /usr/local/harbor/
5.设置harbor配置文件,启动。
[root@localhost7A ~]# grep "^[a-Z]" /usr/local/harbor/harbor.cfg
hostname = harbor.zzhz.com
ui_url_protocol = http
harbor_admin_password = Harbor12345
5.启动
./install.sh
6.docker 命令登录测试
设置要登录harbor服务器的各节点.master和node节点要下载镜像。
默认下harbor是采用的https登录,如果使用http登录.使用此参数:非安全登录方式,--insecure-registry harbor.zzhz.com
[root@localhost7C ~] cat /usr/lib/systemd/system/docker.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry harbor.zzhz.com
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
[root@localhost7C ~]# systemctl daemon-reload && systemctl restart docker
[root@localhost7C ~]# docker login harbor.zzhz.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
#下载
[root@localhost7C ~]# docker pull nginx
[root@localhost7C ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 12766a6745ee 2 weeks ago 141MB
#打标签,baseimage先要创建。
[root@localhost7C ~]# docker tag 12766a6745ee harbor.zzhz.com/baseimage/nginx:latest
#上传Harbor
[root@localhost7C ~]# docker push harbor.zzhz.com/baseimage/nginx:latest
7.web页面登录查看
二、部署haproxy keeplivad高可用反向代理
1.同步时间服务
[root@localhost7B ~]# ntpdate time1.aliyun.com && hwclock -w
开启路由转发
[root@localhost7B ~]# echo 1 > /proc/sys/net/ipv4/ip_forward
2.下载YUM源
[root@localhost7B ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
[root@localhost7B ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
3.安装软件
[root@localhost7B ~]# yum install keepalived haproxy -y
4.配置keepalived
[root@localhost7B ~]# cat /etc/keepalived/keepalived.conf
global_defs {
notification_email {
root@localhost
}
notification_email_from root@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id localhost7B
vrrp_iptables
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
}
vrrp_instance zzhz {
state MASTER
interface eth0
virtual_router_id 51
priority 95
advert_int 2
authentication {
auth_type PASS
auth_pass centos
}
virtual_ipaddress {
192.168.80.222/24 dev eth0 label eth0:1
}
}
5.配置haproxy
[root@localhost7B ~]# cat /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen stats
mode http
bind 0.0.0.0:9999
stats enable
log global
stats uri /haproxy-status
stats auth haadmin:12345
listen k8s-6443
bind 192.168.80.222:6443
mode tcp
balance roundrobin
server 192.168.80.120 192.168.80.120:6443 check inter 2s fall 3 rise 5
server 192.168.80.130 192.168.80.130:6443 check inter 2s fall 3 rise 5
server 192.168.80.140 192.168.80.140:6443 check inter 2s fall 3 rise 5
6.启动服务
[root@localhost7B ~]# systemctl enable keepalived.service haproxy.service
[root@localhost7B ~]# systemctl start keepalived.service
[root@localhost7B ~]# systemctl status keepalived.service
[root@localhost7B ~]# systemctl start haproxy.service
[root@localhost7B ~]# systemctl status haproxy.service
三、部署master
在所有 master 安装指定版本的 kubeadm 、kubelet、kubectl、docker
1.同步时间服务 并关闭防火墙和selinux
[root@localhost7C ~]# ntpdate time1.aliyun.com && hwclock -w
2. /etc/hosts 解析
192.168.80.100 localhost7A.localdomain harbor.zzhz.com
192.168.80.110 localhost7B.localdomain
192.168.80.120 localhost7C.localdomain
192.168.80.130 localhost7D.localdomain
192.168.80.140 localhost7E.localdomain
192.168.80.150 localhost7F.localdomain
192.168.80.160 localhost7G.localdomain
192.168.80.170 localhost7H.localdomain
3.swapoff -a vim /etc/fstab 中的注释掉
4.开启路由转发 net.ipv4.ip_forward =1
5.将桥接的IPv4流量传递到iptables的链
sysctl -w net.bridge.bridge-nf-call-iptables=1
sysctl -w net.bridge.bridge-nf-call-ip6tables=1
永久修改
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF
sysctl -p /etc/sysctl.d/k8s.conf
5.下载YUM源
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
6.安装docker 并启动
yum list docker-ce --showduplicates
yum install docker-ce-19.03.15-3.el7 docker-ce-cli-19.03.15-3.el7
systemctl enable docker && systemctl start docker
7.配置镜像加速器
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://9916w1ow.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload && systemctl restart docker
8.配置k8s的YUM源
[root@localhost7C ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
9.可能需要安装socat
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/socat-1.7.3.2-2.el7.x86_64.rpm
yum install socat-1.7.3.2-2.el7.x86_64.rpm
9.安装kubeadm kubelet kubectl
yum install --nogpgcheck kubeadm-1.17.2-0.x86_64 kubelet-1.17.2-0.x86_64 kubectl-1.17.2-0.x86_64
先要启动,提示错误没关系。
[root@localhost7C ~]# systemctl enable kubelet.service && systemctl start kubelet.service
10.验证当前 kubeadm 版本
[root@localhost7C ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89",
GitTreeState:"clean", BuildDate:"2020-01-18T23:27:49Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
10.查看安装指定版本 k8s 需要的镜像有哪些
[root@localhost7C ~]# kubeadm config images list --kubernetes-version v1.17.2
W0224 11:00:59.345931 29980 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0224 11:00:59.345965 29980 validation.go:28] Cannot validate kubelet config - no validator is available
k8s.gcr.io/kube-apiserver:v1.17.2
k8s.gcr.io/kube-controller-manager:v1.17.2
k8s.gcr.io/kube-scheduler:v1.17.2
k8s.gcr.io/kube-proxy:v1.17.2
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.5
[root@localhost7C ~]#
10.下载所需要版本,推荐提前在 master 节点下载镜像以减少安装等待时间,但是镜像默认使用 Google 的镜像仓
库,所以国内无法直接下载,但是可以通过阿里云的镜像仓库把镜像先提前下载下来,可以避免后期因镜像下载异常而导致 k8s 部署异常。
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.17.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.17.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.5
11.下载所需要版本(脚本)
#!/bin/bash
url=registry.cn-hangzhou.aliyuncs.com/google_containers
version=v1.14.2
images=(`kubeadm config images list --kubernetes-version=$version|awk -F '/' '{print $2}'`)
for imagename in ${images[@]} ; do
docker pull $url/$imagename
docker tag $url/$imagename k8s.gcr.io/$imagename
docker rmi -f $url/$imagename
done
chmod u+x image.sh
./image.sh
12.在三台 master 中任意一台 master 进行集群初始化,而且集群初始化只需要初始化一次。(本例群集)
群集模式:222地址必须有,所以要先配置KA+hp
# kubeadm init --apiserver-advertise-address=192.168.80.120 --apiserver-bind-port=6443 --control-plane-endpoint=192.168.80.222 --kubernetes-version=v1.17.2 --pod-network-cidr=10.10.0.0/16 --service-cidr=10.20.0.0/16 --service-dns-domain=zzhz.local --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --ignore-preflight-errors=swap
单机模式:
kubeadm init --apiserver-advertise-address=192.168.80.120 --apiserver-bind-port=6443 --kubernetes-version=v1.17.2 --pod-network-cidr=10.10.0.0/16 --service-cidr=10.20.0.0/16 --service-dns-domain=linux36.local --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --ignore-preflight-errors=swap
13.完成后的提示,复制到文本中,添加master和node成员需要使用到
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join 192.168.80.222:6443 --token qqjzbt.n9z1zesd7ied1sbe \
--discovery-token-ca-cert-hash sha256:3321b21a12832325852fdab7c10b132b3cca2fb450ac5160de1288ed4ef5700e \
--control-plane --certificate-key 4e5704f14b0fd11d4ad53ab6113825ae28727f9de668ff7f7ccb5151b88f745a
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.80.222:6443 --token qqjzbt.n9z1zesd7ied1sbe \
--discovery-token-ca-cert-hash sha256:3321b21a12832325852fdab7c10b132b3cca2fb450ac5160de1288ed4ef5700e
init初始化成功提示操作步骤
1.创建Kube-config 文件,里面包含 kube-apiserver 地址及相关认证信息,kebectl命令操作的认证。
2.部署网络组件 flannel
3.添加master成员,kubeadm init phase upload-certs --upload-certs命令生成 --certificate-key
4.添加node节点
---------------
14.当前 maste 生成证书用于添加新master控制节点:
# kubeadm init phase upload-certs --upload-certs
[root@localhost7C k8s]# kubeadm init phase upload-certs --upload-certs
I0417 13:10:32.698196 24960 version.go:251] remote version is much newer: v1.23.5; falling back to: stable-1.17
W0417 13:10:34.643215 24960 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0417 13:10:34.643243 24960 validation.go:28] Cannot validate kubelet config - no validator is available
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
4e5704f14b0fd11d4ad53ab6113825ae28727f9de668ff7f7ccb5151b88f745a
14.添加master成员
kubeadm join 192.168.80.222:6443 --token qqjzbt.n9z1zesd7ied1sbe \
--discovery-token-ca-cert-hash sha256:3321b21a12832325852fdab7c10b132b3cca2fb450ac5160de1288ed4ef5700e \
--control-plane --certificate-key 4e5704f14b0fd11d4ad53ab6113825ae28727f9de668ff7f7ccb5151b88f745a
15.kebectl命令配置信息
[root@localhost7C k8s]# mkdir -p $HOME/.kube
[root@localhost7C k8s]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@localhost7C k8s]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@localhost7C k8s]# kubectl get node
NAME STATUS ROLES AGE VERSION
localhost7c.localdomain Ready master 116m v1.17.2
localhost7d.localdomain Ready master 28m v1.17.2
localhost7e.localdomain Ready master 25m v1.17.2
四、部署Node节点设置
0.前期设置的步骤与布署master一样.(1-10步骤)
各需要加入到 k8s master 集群中的 node 节点都要安装 docker kubeadm kubelet,其中kubectl-1.17.2-0.x86_64(可选) ,
因此都要重新执行安装 docker kubeadm kubelet 的步骤,即配置 apt 仓库、配置 docker 加速器、安装命令、启动 kubelet 服务
1.下载所需要版本
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.17.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
2.node节点优化设置
cat >> /etc/security/limits.conf <<EOF
* soft nproc 65535
* hard nproc 65535
* soft nofile 655350
* hard nofile 655350
EOF
sed -i 's#4096#65535#g' /etc/security/limits.d/20-nproc.conf
cat >> /etc/sysctl.conf <<EOF
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 50000
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_max_tw_buckets = 50000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.ip_local_port_range = 1024 65535
vm.swappiness = 0
vm.min_free_kbytes = 524288
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 262144
fs.file-max = 1048576
EOF
sysctl -p
3.安装kubeadm kubelet
[root@localhost7F~]# yum install --nogpgcheck kubeadm-1.17.2-0.x86_64 kubelet-1.17.2-0.x86_64
[root@localhost7F~]# systemctl enable kubelet.service && systemctl start kubelet.service
4.添加node节点
kubeadm join 192.168.80.222:6443 --token qqjzbt.n9z1zesd7ied1sbe \
--discovery-token-ca-cert-hash sha256:3321b21a12832325852fdab7c10b132b3cca2fb450ac5160de1288ed4ef5700e
[root@localhost7C k8s]# kubectl get node
NAME STATUS ROLES AGE VERSION
localhost7c.localdomain NotReady master 143m v1.17.2
localhost7d.localdomain NotReady master 54m v1.17.2
localhost7e.localdomain NotReady master 52m v1.17.2
localhost7f.localdomain NotReady <none> 16s v1.17.2
localhost7g.localdomain NotReady <none> 24s v1.17.2
localhost7h.localdomain NotReady <none> 34s v1.17.2
五、部署网络组件 flannel
1.安装网络插件,一般的网络无法访问quay.io,可以曲线救国,找国内的镜像源,或者从docker hub上拉取flannel的镜像手动拉取flannel镜像,在集群的所有机器上操作
# 手动拉取flannel的docker镜像
[root@localhost7C k8s]#docker pull easzlab/flannel:v0.12.0-amd64
# 修改镜像名称
[root@localhost7C k8s]#docker tag ff281650a721 harbor.zzhz.com/baseimage/flannel:v0.12.0-amd64
[root@localhost7C k8s]#docker push harbor.zzhz.com/baseimage/flannel:v0.12.0-amd64
2.下载flannel资源配置清单文件 wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
修改里面的镜像地址 和与--pod-network-cidr=10.10.0.0/16同一网段地址
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.10.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-amd64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: harbor.zzhz.com/baseimage/flannel:v0.12.0-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: harbor.zzhz.com/baseimage/flannel:v0.12.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- arm64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.12.0-arm64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-arm64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- arm
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.12.0-arm
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-arm
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-ppc64le
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- ppc64le
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.12.0-ppc64le
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-ppc64le
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-s390x
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- s390x
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.12.0-s390x
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-s390x
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
View Code
3.安装flannel
# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged configured
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
3.[root@localhost7C k8s]# kubectl get node
NAME STATUS ROLES AGE VERSION
localhost7c.localdomain NotReady master 143m v1.17.2
localhost7d.localdomain NotReady master 54m v1.17.2
localhost7e.localdomain NotReady master 52m v1.17.2
localhost7f.localdomain NotReady <none> 16s v1.17.2
localhost7g.localdomain NotReady <none> 24s v1.17.2
localhost7h.localdomain NotReady <none> 34s v1.17.2
4.查看日志,说明docker:网络插件未就绪:cni配置未初始化,
[root@localhost7C ~]# kubectl describe nodes localhost7c.localdomain
luginNotReady message:docker: network plugin is not ready: cni config uninitialized
[root@localhost7C ~]# systemctl status kubelet.service
[root@localhost7C ~]# journalctl -f -u kubelet.service
-- Logs begin at 日 2023-02-26 21:54:47 CST. --
2月 27 10:57:18 localhost7D.localdomain kubelet[10247]: "type": "portmap",
2月 27 10:57:18 localhost7D.localdomain kubelet[10247]: "capabilities": {
2月 27 10:57:18 localhost7D.localdomain kubelet[10247]: "portMappings": true
2月 27 10:57:18 localhost7D.localdomain kubelet[10247]: }
2月 27 10:57:18 localhost7D.localdomain kubelet[10247]: }
2月 27 10:57:18 localhost7D.localdomain kubelet[10247]: ]
2月 27 10:57:18 localhost7D.localdomain kubelet[10247]: }
2月 27 10:57:18 localhost7D.localdomain kubelet[10247]: : [failed to find plugin "flannel" in path [/opt/cni/bin]]
2月 27 10:57:18 localhost7D.localdomain kubelet[10247]: W0227 10:57:18.597526 10247 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d
2月 27 10:57:21 localhost7D.localdomain kubelet[10247]: E0227 10:57:21.727513 10247 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
下载插件:(在k8s 1.0.0版本后CNI Plugins中没有flannel,需自行下载)
https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
tar xvf cni-plugins-linux-amd64-v0.8.6.tgz
scp /opt/cni/bin/flannel 192.168.80.150:/opt/cni/bin/
安装插件(在k8s 1.0.0版本之前)
yum install kubernetes-cni -y
5.查看集群的node状态,安装完网络工具之后,只有显示如下状态,所有节点全部都Ready好了之后才能继续后面的操作
[root@localhost7C k8s]# kubectl get node
NAME STATUS ROLES AGE VERSION
localhost7c.localdomain Ready master 2d23h v1.17.2
localhost7d.localdomain Ready master 2d23h v1.17.2
localhost7e.localdomain Ready master 2d23h v1.17.2
localhost7f.localdomain Ready <none> 2d23h v1.17.2
localhost7g.localdomain Ready <none> 2d23h v1.17.2
六、部署 web 服务 dashboard
1.设置要登录harbor服务器的各节点.
cat /usr/lib/systemd/system/docker.service
默认下harbor是采用的https登录,如果使用http登录.使用此参数:非安全登录方式,--insecure-registry harbor.zzhz.com
[root@localhost7A ~] cat /usr/lib/systemd/system/docker.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry harbor.zzhz.com
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
[root@localhost7A ~]# systemctl daemon-reload && systemctl restart docker
2.下载dashboard和metrics-scraper镜像,
[root@localhost7A ~]# docker pull kubernetesui/dashboard:v2.0.0-rc6
v2.0.0-rc6: Pulling from kubernetesui/dashboard
1f45830e3050: Pull complete
Digest: sha256:61f9c378c427a3f8a9643f83baa9f96db1ae1357c67a93b533ae7b36d71c69dc
Status: Downloaded newer image for kubernetesui/dashboard:v2.0.0-rc6
docker.io/kubernetesui/dashboard:v2.0.0-rc6
[root@localhost7A ~]# docker pull kubernetesui/metrics-scraper:v1.0.3
v1.0.3: Pulling from kubernetesui/metrics-scraper
75d12d4b9104: Pull complete
fcd66fda0b81: Pull complete
53ff3f804bbd: Pull complete
Digest: sha256:40f1d5785ea66609b1454b87ee92673671a11e64ba3bf1991644b45a818082ff
Status: Downloaded newer image for kubernetesui/metrics-scraper:v1.0.3
docker.io/kubernetesui/metrics-scraper:v1.0.3
[root@localhost7A ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
kubernetesui/dashboard v2.0.0-rc6 cdc71b5a8a0e 2 years ago 221MB
kubernetesui/metrics-scraper v1.0.3 3327f0dbcb4a 2 years ago 40.1MB
3.打标和上传到Harbor的baseimage项目中.
[root@localhost7A ~]docker tag cdc71b5a8a0e harbor.zzhz.com/baseimage/dashboard:v2.0.0-rc6
[root@localhost7A ~]docker tag 3327f0dbcb4a harbor.zzhz.com/baseimage/metrics-scraper:v1.0.3
[root@localhost7A ~]# docker login harbor.zzhz.com
[root@localhost7A ~]docker push harbor.zzhz.com/baseimage/metrics-scraper:v1.0.3
[root@localhost7A ~]docker push harbor.zzhz.com/baseimage/dashboard:v2.0.0-rc6
4.设置yaml文件
[root@localhost7C k8s]# ls
admin-user.yml #用户文件
dash_board-2.0.0-rc6.yml
[root@localhost7C k8s]# vim dash_board-2.0.0-rc6.yml #设置端口和镜像地址
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30002 #设置端口
selector:
k8s-app: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kubernetes-dashboard
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
namespace: kubernetes-dashboard
type: Opaque
data:
csrf: ""
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-key-holder
namespace: kubernetes-dashboard
type: Opaque
---
kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
namespace: kubernetes-dashboard
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
rules:
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster", "dashboard-metrics-scraper"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: harbor.zzhz.com/baseimage/dashboard:v2.0.0-rc6 #设置image
imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
- --namespace=kubernetes-dashboard
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
nodeSelector:
"beta.kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: dashboard-metrics-scraper
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
spec:
containers:
- name: dashboard-metrics-scraper
image: harbor.zzhz.com/baseimages/metrics-scraper:v1.0.3 #设置image
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
serviceAccountName: kubernetes-dashboard
nodeSelector:
"beta.kubernetes.io/os": linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}
View Code
cat admin-user.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
View Code
[root@localhost7C k8s]# kubectl apply -f dash_board-2.0.0-rc6.yml -f admin-user.yml
5.获取用户登录 token
[root@localhost7C k8s]# kubectl get secret -A | grep admin-user
kubernetes-dashboard admin-user-token-4htwp kubernetes.io/service-account-token 3 3m10s
[root@localhost7C k8s]# kubectl describe secret -n kubernetes-dashboard admin-user-token-4htwp
Name: admin-user-token-4htwp
Namespace: kubernetes-dashboard
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-user
kubernetes.io/service-account.uid: d7880161-6c5b-4b94-8dff-076a4f27a393
Type: kubernetes.io/service-account-token
Data
====
namespace: 20 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjJwWTNxbVhFRXh4ZzR3V2EwakRleThrQ2U1c1A3WlZGekRFcXZ0UU8xN0UifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLTRodHdwIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkNzg4MDE2MS02YzViLTRiOTQtOGRmZi0wNzZhNGYyN2EzOTMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.RIZuIdo1ne-nXnCF9qDMMwGtcv6rrMYALrdnCYv_oeC3dTO14r5-RpVNycD-2G8IV3SQKUrqE6RaUiLXmZzQUpIDOqO7SdECIbHEix33nAE2qB0KAmk6lMbB0z53B5SG_2dS4H-YheDCAKcnqRMi00agjoTnL3X7-ehTgAuVNBugBdOha2RvxLDCmHA3JUvjM6Aeoj0715nsD2pA3l9VzQ7eFcbN1dbacri6H0sZ9hEWdiHCUnj0cecR_5FQhySnH5gcIrBbTZSmk9Gp8U4sB82uI47cmVV7JKlTd1W5VvXX8HnsLB9cDQXzomg59C-QuGdAhGjB_3L2m7tB8dVoRQ
ca.crt: 1025 bytes
[root@localhost7C k8s]#
6.使用token测试登录 dashboard: https://192.168.80.120:30002/#/login
七、升级k8s版本
1.升级master:
1.1先安装指定版本的kubeadm
[root@localhost7D ~]# kubeadm version
[root@localhost7D ~]# yum install kubeadm-1.17.17
[root@localhost7D ~]# kubeadm version
1.2.查看升级计划kubeadm upgrade plan
[root@localhost7C ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.17.2
[upgrade/versions] kubeadm version: v1.17.17
I0227 14:54:06.681442 29772 version.go:251] remote version is much newer: v1.26.1; falling back to: stable-1.17
[upgrade/versions] Latest stable version: v1.17.17
[upgrade/versions] Latest version in the v1.17 series: v1.17.17
Upgrade to the latest version in the v1.17 series:
COMPONENT CURRENT AVAILABLE
API Server v1.17.2 v1.17.17
Controller Manager v1.17.2 v1.17.17
Scheduler v1.17.2 v1.17.17
Kube Proxy v1.17.2 v1.17.17
CoreDNS 1.6.5 1.6.5
Etcd 3.4.3 3.4.3-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.17.17
1.3.kubeadm upgrade apply v1.17.17(升级过程中会下载镜像,可以先下载)
[root@localhost7C ~]# kubeadm config images list --kubernetes-version v1.17.17
k8s.gcr.io/kube-apiserver:v1.17.17
k8s.gcr.io/kube-controller-manager:v1.17.17
k8s.gcr.io/kube-scheduler:v1.17.17
k8s.gcr.io/kube-proxy:v1.17.17
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.5
先下载镜像
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.17
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.17.17
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.17
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.17.17
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.5
1.4.更新k8s, 后查看下pod使用的镜像。
[root@localhost7C ~]# kubeadm upgrade apply v1.17.17
1.5.升级 kubelet=1.17.17 kubectl=1.17.17
[root@localhost7C ~]# yum install kubelet-1.17.17 kubectl-1.17.17
[root@localhost7C ~]# systemctl daemon-reload && systemctl restart kubelet
[root@localhost7C ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
localhost7c.localdomain Ready master 3d3h v1.17.17(kubelet版本)
localhost7d.localdomain Ready master 3d3h v1.17.17
localhost7e.localdomain Ready master 3d3h v1.17.17
2.升级各node节点:
2.1.kubeadm需要升级吗?
[root@localhost7G ~]# yum install kubeadm-1.17.17 -y
2.2.各node节点配置文件:
[root@localhost7G ~]# kubeadm upgrade node --kubelet-version 1.17.17
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade] Skipping phase. Not a control plane node.
[upgrade] Using kubelet config version 1.17.17, while kubernetes-version is v1.17.17
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
2.3.升级 kubelet=1.17.17 ( kubectl=1.17.17 node一般不安装kubectl,不用升级。)
[root@localhost7F ~]# yum install kubelet-1.17.17
[root@localhost7F ~]# systemctl daemon-reload && systemctl restart kubelet
[root@localhost7C ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
localhost7c.localdomain Ready master 3d3h v1.17.17(kubelet版本)
localhost7d.localdomain Ready master 3d3h v1.17.17
localhost7e.localdomain Ready master 3d3h v1.17.17
localhost7f.localdomain Ready <none> 3d3h v1.17.17
localhost7g.localdomain Ready <none> 3d3h v1.17.17
八、项目实例:运行 Nginx+Tomcat,实现动静分离:
客户端访问网站流程:
1.客户端访问到公网IP到HAproxy
2.HAproxy到其中某一node宿主机上监听的端口(nodeport)
3.nodeport转发到service,service根据标签转发到nginx(service指的是网络,筛选器)
4.nginx如有后端,这里配置没有写IP,是标签名,根据kube-dns解析得到tomcat的label标签.
1.项目目录结构
[root@localhost7C k8s]# tree nginx-tomcat/
nginx-tomcat/
├── nginx-dockerfile
│ ├── conf
│ │ └── default.conf #nginx配置文件
│ └── Dockerfile #制作镜像文件
├── nginx.yml #启动nginx容器文件
│
├── tomcat-dockerfile
│ ├── app
│ │ └── index.html #nginx配置文件
│ └── Dockerfile #制作镜像文件
└── tomcat.yml #启动tomcat容器文件
2.下载镜像
[root@localhost7C k8s]# docker pull tomcat
[root@localhost7C k8s]# docker pull nginx
3.查看和制作tomcat镜像
[root@localhost7C tomcat-dockerfile]# cat app/index.html
tomcat app images web page
[root@localhost7C tomcat-dockerfile]# cat Dockerfile
FROM tomcat
ADD ./app /usr/local/tomcat/webapps/app/
制作镜像
[root@localhost7C tomcat-dockerfile]# docker build -t harbor.zzhz.com/baseimage/tomcat:app1 .
测试tomcat是否能访问(可过)
[root@localhost7C tomcat-dockerfile]# docker run -it --rm -p 8080:8080 harbor.zzhz.com/baseimage/tomcat:app1
#curl http://192.168.80.120:8080/app/
tomcat app images web page
上传到harbor
[root@localhost7C k8s]# docker push harbor.zzhz.com/baseimage/tomcat:app1
4.查看和制作nginx镜像,
[root@localhost7C nginx-dockerfile]# cat conf/default.conf
server {
listen 80;
listen [::]:80;
server_name localhost;
#access_log /var/log/nginx/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
#添加这里
location /app {
proxy_pass http://magedu-tomcat-service;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
[root@localhost7C nginx-dockerfile]# cat Dockerfile
FROM nginx
ADD ./conf/default.conf /etc/nginx/conf.d/
#制作镜像
[root@localhost7C nginx-dockerfile]# docker build -t harbor.zzhz.com/baseimage/nginx:v1 .
#如果要测试:nginx定义了proxy_pass,后端没启动会提示错误。)
[root@localhost7C nginx-dockerfile]#docker run -it --rm -p 80:80 harbor.zzhz.com/baseimage/nginx:v1
上传到harbor
[root@localhost7C nginx-tomcat]# docker push harbor.zzhz.com/baseimage/nginx:v1
5.查看tomcat和nginx的yaml文件,并运行
#tomcat文件
[root@localhost7C nginx-tomcat]# cat tomcat.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deployment
labels:
app: tomcat
spec:
replicas: 1
selector:
matchLabels:
app: tomcat
template:
metadata:
labels:
app: tomcat
spec:
containers:
- name: tomcat
image: harbor.zzhz.com/baseimage/tomcat:app1
ports:
- containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
labels:
app: magedu-tomcat-service-label
name: magedu-tomcat-service
namespace: default
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
nodePort: 30005
selector:
app: tomcat
#nginx文件
[root@localhost7C nginx-tomcat]# cat nginx.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: harbor.zzhz.com/baseimage/nginx:v1
ports:
- containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
labels:
app: magedu-nginx-service-label
name: magedu-nginx-service
namespace: default
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
nodePort: 30004
selector:
app: nginx
6.运行容器
[root@localhost7C nginx-tomcat]# kubectl apply -f tomcat.yml -f nginx.yml
7.通过 HAProxy 实现高可用反向代理:
基于 haproxy 和 keepalived 实现高可用的反向代理,并访问到运行在 kubernetes集群中业务 Pod。
[root@localhost7B ~]# cat /etc/keepalived/keepalived.conf
....
....
virtual_ipaddress {
192.168.80.222/24 dev eth0 label eth0:1
192.168.80.223/24 dev eth0 label eth0:2
}
[root@localhost7B ~]# cat /etc/haproxy/haproxy.conf
....
....
listen k8s-6443
bind 192.168.80.222:6443
mode tcp
balance roundrobin
server 192.168.80.110 192.168.80.110:6443 check inter 2s fall 3 rise 5
server 192.168.80.120 192.168.80.120:6443 check inter 2s fall 3 rise 5
server 192.168.80.130 192.168.80.130:6443 check inter 2s fall 3 rise 5
listen nginx-80
bind 192.168.80.223:80
mode tcp
balance roundrobin
server 192.168.80.150 192.168.80.150:30004 check inter 2s fall 3 rise 5
server 192.168.80.160 192.168.80.160:30004 check inter 2s fall 3 rise 5
8.测试:
[root@localhost7A harbor]# curl 192.168.80.223
<h1>Welcome to nginx!</h1>
[root@localhost7A harbor]# curl 192.168.80.223/app/index.html
tomcat app images web page
9.业务逻辑说明:
[root@localhost7C nginx-tomcat]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.20.0.1 <none> 443/TCP 4d4h
magedu-nginx-service NodePort 10.20.109.183 <none> 80:30004/TCP 52m
magedu-tomcat-service NodePort 10.20.85.117 <none> 80:30005/TCP 58m
[root@localhost7C nginx-tomcat]# kubectl describe service magedu-tomcat-service
Name: magedu-tomcat-service
Namespace: default
Labels: app=magedu-tomcat-service-label
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"magedu-tomcat-service-label"},"name":"magedu-tomcat-serv...
Selector: app=tomcat
Type: NodePort
IP: 10.20.85.117 #地址
Port: http 80/TCP
TargetPort: 8080/TCP
NodePort: http 30005/TCP
Endpoints: 10.10.4.6:8080 #后端服务
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
进入到 nginx Pod
[root@localhost7C ~]# kubectl exec -it nginx-deployment-5999b8f5d6-5pz96 bash
# cat /etc/issue
Debian GNU/Linux 9 \n \l
更新软件源并安装基础命令
# apt update
# apt install procps vim iputils-ping net-tools curl
测试 service 解析
root@nginx-deployment-5999b8f5d6-5pz96:/# ping magedu-tomcat-service
PING magedu-tomcat-service.default.svc.zzhz.local (10.20.85.117) 56(84) bytes of data.
测试在 nginx Pod 通过 tomcat Pod 的 service 域名访问:
root@nginx-deployment-5999b8f5d6-5pz96:/# curl magedu-tomcat-service.default.svc.zzhz.local/app/index.html
tomcat app images web page