Kubernetnes部署之控制篇

 

接着地基篇的部署，我们开始搞控制节点

四、Master部署

K8s控制节点:我们以flannel为分界中心，运行apiServer的为控制节点

运行的组件包括：

kube-apiserver kube-controller-manager kube-scheduler

 

1．准备工作

• 安装可执行程序

==> /data/k8s/bin/{ kube-apiserver | kube-controller-master | kube-scheduler }

• 配置数据存储目录

==> /data/kubernetes/{ kube-apiserver | kube-controller-master | kube-scheduler }

• 获取认证key工具==> /etc/kubernetes/cert/

# apiServer服务需要认证各种key # mkidr /etc/kubernetes/cert/ curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl                #生成证书工具 curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson           #通过json文件生成证书 curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo   #证书查看工具 chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo   # cfssl gencert -initca ca-csr.json | cfssljson -bare ca - # cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

 

2．部署ApiServer服务

API接口服务：Apiserver

# 服务启动配置文件（需要自己的认证key，还是2个） # /etc/systemd/system/kube-apiserver.service   [Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target   [Service] WorkingDirectory=/data/kubernetes/kube-apiserver ExecStart=/data/k8s/bin/kube-apiserver \   --bind-address=192.168.200.54 \   --advertise-address=192.168.200.54 \   --default-not-ready-toleration-seconds=360 \   --default-unreachable-toleration-seconds=360 \   --feature-gates=DynamicAuditing=true \   --max-mutating-requests-inflight=2000 \   --max-requests-inflight=4000 \   --default-watch-cache-size=200 \   --delete-collection-workers=2 \   --encryption-provider-config=/data/k8s/conf/kube-apiserver/encryption-config.yaml \   --etcd-cafile=/data/kubernetes/cert/ca.pem \   --etcd-certfile=/data/k8s/cert/server/server.pem \   --etcd-keyfile=/data/k8s/cert/server/server-key.pem \ --etcd-servers=https://192.168.200.136:2379,https://192.168.200.137:2379,https://192.168.200.138:2379\   --secure-port=6443 \   --tls-cert-file=/data/k8s/cert/kube-apiserver/apiserver.pem \   --tls-private-key-file=/data/k8s/cert/kube-apiserver/apiserver-key.pem \   --audit-dynamic-configuration \   --audit-log-maxage=15 \   --audit-log-maxbackup=3 \   --audit-log-maxsize=100 \   --audit-log-truncate-enabled \   --audit-log-path=/data/kubernetes/kube-apiserver/logs/audit.log \   --audit-policy-file=/data/k8s/conf/kube-apiserver/audit-policy.yaml \   --profiling \   --anonymous-auth=false \   --client-ca-file=/etc/kubernetes/cert/ca.pem \   --token-auth-file=/etc/kubernetes/token.csv \   --enable-bootstrap-token-auth \   --requestheader-allowed-names="aggregator" \   --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \   --requestheader-extra-headers-prefix="X-Remote-Extra-" \   --requestheader-group-headers=X-Remote-Group \   --requestheader-username-headers=X-Remote-User \   --authorization-mode=RBAC,Node\   --runtime-config=api/all=true \   --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \   --allow-privileged=true \   --apiserver-count=3 \   --event-ttl=168h \   --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \   --kubelet-client-certificate=/data/k8s/cert/kube-apiserver/apiserver.pem \   --kubelet-client-key=/data/k8s/cert/kube-apiserver/apiserver-key.pem \   --kubelet-https=true \   --kubelet-timeout=10s \   --proxy-client-cert-file=/data/k8s/cert/proxy-client/proxy-client.pem \   --proxy-client-key-file=/data/k8s/cert/proxy-client/proxy-client-key.pem \   --service-account-key-file=/etc/kubernetes/cert/ca.pem \   --service-cluster-ip-range=10.17.0.0/16 \   --service-node-port-range=30000-32767 \   --log-dir=/data/kubernetes/kube-apiserver/svrlogs \   --logtostderr=false \   --v=2   Restart=on-failure RestartSec=10 Type=notify LimitNOFILE=65536   [Install] WantedBy=multi-user.target

3．部署CtlManager服务

控制器服务：Controller-manager

kube-controller-manager 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-controller-manager 证书   1)生成自用的公钥私钥key，创建一个controller-manager-csr.json {   "CN": "system:kube-controller-manager",   "hosts": [      "127.0.0.1",      "192.168.200.54",      "192.168.200.55",      "192.168.200.56",      "gray-master01",      "gray-master02",      "gray-master03",      "kubernetes",      "kubernetes.default",      "kubernetes.default.svc",      "kubernetes.default.svc.cluster",      "kubernetes.default.svc.cluster.local"   ],   "key": {     "algo": "rsa",     "size": 2048   },   "names": [     {       "C": "CN",       "ST": "Hangzhou",       "L": "Zhejiang",       "O": "system:kube-controller-manager",       "OU": "System"     }   ] }   2)生成 controller-manager.csr  controller-manager.pem  controller-manager-key.pem文件 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \ -profile=kubernetes controller-manager-csr.json | cfssljson -bare controller-manager   3)执行一些命令 ==》 设置集群参数   kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem \ --embed-certs=true --server=https://192.168.200.54:6443 \ --kubeconfig=kube-controller-manager.kubeconfig 注意：https://192.168.200.55:6443,https://192.168.200.56:6443 提示“Cluster "kubernetes" set.”（各节点独自执行指定的--server地址）   ==》 设置控制器认证参数   kubectl config set-credentials system:kube-controller-manager --embed-certs=true \ --client-certificate=controller-manager.pem  --client-key=controller-manager-key.pem \ --kubeconfig=kube-controller-manager.kubeconfig 提示“User "system:kube-controller-manager" set.”   ==》 设置控制器上下文   kubectl config set-context system:kube-controller-manager --cluster=kubernetes  \ --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig 提示“Context "system:kube-controller-manager" created”   ==》 开启默认使用   kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig 提示“Switched to context "system:kube-controller-manager".”   4)服务启动配置文件（需要自己的认证key）,启动并设置自启动 # /etc/systemd/system/kube-controller-manager.service   [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes   [Service] WorkingDirectory=/data/kubernetes/kube-controller-manager ExecStart=/data/k8s/bin/kube-controller-manager \   --kubeconfig=/data/k8s/conf/kube-controller-manager/kube-controller-manager.kubeconfig \   --bind-address=127.0.0.1 \   --secure-port=10252 \   --leader-elect=true \ --port=0 \   --profiling \   --cluster-name=kubernetes \   --controllers=*,bootstrapsigner,tokencleaner \   --kube-api-qps=1000 \   --kube-api-burst=2000 \   --use-service-account-credentials=true \   --concurrent-service-syncs=2 \   --tls-cert-file=/data/k8s/cert/kube-controller-manager/controller-manager.pem \   --tls-private-key-file=/data/k8s/cert/kube-controller-manager/controller-manager-key.pem \   --authentication-kubeconfig=/data/k8s/conf/kube-controller-manager/kube-controller-manager.kubeconfig \   --client-ca-file=/etc/kubernetes/cert/ca.pem \   --requestheader-allowed-names="" \(没有这个参数，去掉)   --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \   --requestheader-extra-headers-prefix="X-Remote-Extra-" \   --requestheader-group-headers=X-Remote-Group \   --requestheader-username-headers=X-Remote-User \   --authorization-kubeconfig=/data/k8s/conf/kube-controller-manager/kube-controller-manager.kubeconfig \   --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \   --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \   --experimental-cluster-signing-duration=876000h \   --horizontal-pod-autoscaler-use-rest-clients=true \   --horizontal-pod-autoscaler-sync-period=10s \   --concurrent-deployment-syncs=10 \   --concurrent-gc-syncs=30 \   --node-cidr-mask-size=24 \   --service-cluster-ip-range=10.18.0.0/16 \   --pod-eviction-timeout=6m \   --terminated-pod-gc-threshold=10000 \   --root-ca-file=/etc/kubernetes/cert/ca.pem \   --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \   --feature-gates=RotateKubeletServerCertificate=true \   --log-dir=/data/kubernetes/kube-controller-manager/svrlogs \   --logtostderr=false \   --v=2   Restart=on-failure RestartSec=5 LimitNOFILE=65536   [Install] WantedBy=multi-user.target   5)权限验证查看 kubectl describe clusterrole system:kube-controller-manager kubectl get clusterrole|grep controller kubectl get cs

4．部署Scheduler服务

提供调度服务：Schedler

kube-scheduler使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-scheduler 证书   1)生成自用的公钥私钥key，创建一个scheduler-csr.json {   "CN": "system:kube-scheduler",   "hosts": [      "127.0.0.1",      "192.168.200.54",      "192.168.200.55",      "192.168.200.56",      "kubernetes",      "kubernetes.default",      "kubernetes.default.svc",      "kubernetes.default.svc.cluster",      "kubernetes.default.svc.cluster.local"   ],   "key": {     "algo": "rsa",     "size": 2048   },   "names": [     {       "C": "CN",       "ST": "Hangzhou",       "L": "Zhejiang",       "O": "system:kube-scheduler",       "OU": "System"     }   ] }   2)生成 scheduler.csr  scheduler.pem  scheduler-key.pem cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \ -profile=kubernetes scheduler-csr.json | cfssljson -bare scheduler   3)执行一些命令 ==》 设置集群参数   kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem \ --embed-certs=true --server=https://192.168.200.54:6443 --kubeconfig=kube-scheduler.kubeconfig 注意：https://192.168.200.55:6443,https://192.168.200.56:6443 提示“Cluster "kubernetes" set.”（各节点独自执行指定的--server地址）   ==》 设置调度器认证参数   kubectl config set-credentials system:kube-scheduler --client-certificate=scheduler.pem \ --client-key=scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig 提示“User "system:kube-scheduler" set.”   ==》 设置调度器上下文   kubectl config set-context system:kube-scheduler --cluster=kubernetes  \ --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig 提示“Context "system:kube-scheduler" created.”   ==》 开启默认上下文 kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig 提示“Switched to context "system:kube-scheduler".”   4)自建配置文件（kube-scheduler.yaml） cat >kube-scheduler-config.yaml <<EOF apiVersion: kubescheduler.config.k8s.io/v1alpha1 kind: KubeSchedulerConfiguration bindTimeoutSeconds: 600 clientConnection:   burst: 200   kubeconfig: "/data/k8s/conf/kube-scheduler/kube-scheduler.kubeconfig"   qps: 100 enableContentionProfiling: false enableProfiling: true hardPodAffinitySymmetricWeight: 1 healthzBindAddress: 127.0.0.1:10251 leaderElection:   leaderElect: true #选举集群 metricsBindAddress: 192.168.200.54:10251 EOF   5)服务启动配置文件（需要自己的认证key）,，启动并设置自启动 # /etc/systemd/system/kube-scheduler.service   [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/GoogleCloudPlatform/kubernetes   [Service] WorkingDirectory=/data/kubernetes/kube-scheduler ExecStart=/data/k8s/bin/kube-scheduler \   --config=/data/k8s/conf/kube-scheduler/kube-scheduler-config.yaml \   --bind-address=127.0.0.1 \   --secure-port=10259 \   --port=0 \   --leader-elect=true \   --tls-cert-file=/data/k8s/cert/kube-scheduler/scheduler.pem \   --tls-private-key-file=/data/k8s/cert/kube-scheduler/scheduler-key.pem \   --authentication-kubeconfig=/data/k8s/conf/kube-scheduler/kube-scheduler.kubeconfig \   --client-ca-file=/etc/kubernetes/cert/ca.pem \   --requestheader-allowed-names="" \(没有这个参数，去掉)   --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \   --requestheader-extra-headers-prefix="X-Remote-Extra-" \   --requestheader-group-headers=X-Remote-Group \   --requestheader-username-headers=X-Remote-User --authorization-kubeconfig=/data/k8s/conf/kube-scheduler/kube-scheduler.kubeconfig \ --log-dir=/data/kubernetes/kube-scheduler/svrlogs \   --logtostderr=false \   --v=2   Restart=always RestartSec=5 StartLimitInterval=0   [Install] WantedBy=multi-user.target   6)权限验证查看 kubectl get endpoints kube-scheduler --namespace=kube-system  -o yaml kubectl get cs

5．说明其他指令

• kube-controller-manager 使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-controller-manager 证书
• 设置集群入口

==》 kubectl config set-cluster kubernetes --embed-certs=true \

--certificate-authority=/home/kubernetes/cert/ca.pem \

--server=https://192.168.200.57:6443 --kubeconfig=kube-controller-manager.kubeconfig  

提示“Cluster "kubernetes" set.”生成 kube-controller-manager.kubeconfig 文件

• 设置控制器认证参数

==》 kubectl config set-credentials system:kube-controller-manager --embed-certs=true

--client-certificate=kube-controller-manager.pem \

--client-key=kube-controller-manager-key.pem \

 --kubeconfig=kube-controller-manager.kubeconfig

提示“User "system:kube-controller-manager" set.”用户生成

• 设置控制器上下文

==》 kubectl config set-context system:kube-controller-manager --cluster=kubernetes \

--user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

提示“Context "system:kube-controller-manager" created” 上下文生成

• 开启使用

==》 kubectl config use-context \

system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

提示“Switched to context "system:kube-controller-manager".”

• kube-scheduler使用 kubeconfig 文件访问 apiserver，该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-scheduler 证书
• 设置集群参数

==》 kubectl config set-cluster kubernetes --embed-certs=true \

--certificate-authority=/home/kubernetes/cert/ca.pem

--server=https://192.168.200.57:6443 --kubeconfig=kube-scheduler.kubeconfig  

提示“Cluster "kubernetes" set.”集群入口创建，生成 kube-scheduler.kubeconfig 文件

• 设置调度器认证参数

==》 kubectl config set-credentials system:kube-scheduler --embed-certs=true \

--client-certificate=kube-scheduler.pem \

--client-key=kube-scheduler-key.pem  --kubeconfig=kube-scheduler.kubeconfig

提示“User "system:kube-scheduler" set.”用户生成

• 设置调度器上下文

==》 kubectl config set-context system:kube-scheduler --cluster=kubernetes  \

--user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

提示“Context "system:kube-scheduler" created.”上下文生成

• 开启使用

==》 kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

提示“Switched to context "system:kube-scheduler".”