k8s实践7:etcd实践操作记录

原创

am2012 2019-04-18 15:55:12 博主文章分类：k8s ©著作权

文章标签 kubernetes etcd 数据备份恢复 etcd删除pod 文章分类 运维

©著作权归作者所有：来自51CTO博客作者am2012的原创作品，请联系作者获取转载授权，否则将追究法律责任

etcd实践操作记录

etcd是kubernetes的核心中的核心.kubernetes所有数据都存储在etcd,怎么检索删除etcd数据怎么备份恢复etcd数据?

1.etcd的api版本说明

flannel操作etcd使用的是v2的API,而kubernetes操作etcd使用的v3的API,所以在下面我们执行etcdctl的时候需要设置ETCDCTL_API环境变量,该变量默认值为2.

Etcd V2和V3之间的数据结构完全不同,互不兼容,也就是说使用V2版本的API创建的数据只能使用V2的API访问,V3的版本的API创建的数据只能使用V3的API访问.
这就造成我们访问etcd中保存的flannel的数据需要使用etcdctl的V2版本的客户端,而访问kubernetes的数据需要设置ETCDCTL_API=3环境变量来指定V3版本的API.

2.基础操作记录

v2 api的操作记录

kubernetes默认使用的api

etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem --version
etcdctl version: 3.3.7
API version: 2

检索节点信息,etcd使用的是leader模式.

etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem member list
1c83ad9421d77430: name=k8s-node3 peerURLs=https://192.168.174.130:2380 clientURLs=https://192.168.174.130:2379 isLeader=false
5eec694677c3c515: name=k8s-node2 peerURLs=https://192.168.174.129:2380 clientURLs=https://192.168.174.129:2379 isLeader=true
65f8d952bfce7d85: name=k8s-node1 peerURLs=https://192.168.174.128:2380 clientURLs=https://192.168.174.128:2379 isLeader=false

api v2检索etcd的所有数据,存储的是flannel网络信息

etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem ls 
/kubernetes

etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem ls /kubernetes
/kubernetes/network
etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem ls /kubernetes/network
/kubernetes/network/config
/kubernetes/network/subnets

检索flannel网络信息,可见使用的是vxlan模式

etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem get /kubernetes/network/config
{"Network":"172.30.0.0/16",
"SubnetLen": 24, "Backend": {"Type": "vxlan"}}

对比下,下面是写入etcd网络信息时的命令

 etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'",
> "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'

检索flannel节点子网分配情况,一个节点分配一个子网

etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem ls /kubernetes/network/subnets
/kubernetes/network/subnets/172.30.75.0-24
/kubernetes/network/subnets/172.30.49.0-24
/kubernetes/network/subnets/172.30.77.0-24

3.v3 api的操作记录

操作前设置api,export ETCDCTL_API=3

注意api v3的操作命令和v2不一样.

设置api,检索版本

export ETCDCTL_API=3

etcdctl version
etcdctl version: 3.3.7
API version: 3.3

etcdctl命令参考

etcdctl -h
NAME:
	etcdctl - A simple command line client for etcd3.

USAGE:
	etcdctl

VERSION:
	3.3.7

API VERSION:
	3.3


COMMANDS:
	get			Gets the key or a range of keys
	put			Puts the given key into the store
	del			Removes the specified key or range of keys [key, range_end)
	txn			Txn processes all the requests in one transaction
	compaction		Compacts the event history in etcd
	alarm disarm		Disarms all alarms
	alarm list		Lists all alarms
	defrag			Defragments the storage of the etcd members with given endpoints
	endpoint health		Checks the healthiness of endpoints specified in `--endpoints` flag
	endpoint status		Prints out the status of endpoints specified in `--endpoints` flag
	endpoint hashkv		Prints the KV history hash for each endpoint in --endpoints
	move-leader		Transfers leadership to another etcd cluster member.
	watch			Watches events stream on keys or prefixes
	version			Prints the version of etcdctl
	lease grant		Creates leases
	lease revoke		Revokes leases
	lease timetolive	Get lease information
	lease list		List all active leases
	lease keep-alive	Keeps leases alive (renew)
	member add		Adds a member into the cluster
	member remove		Removes a member from the cluster
	member update		Updates a member in the cluster
	member list		Lists all members in the cluster
	snapshot save		Stores an etcd node backend snapshot to a given file
	snapshot restore	Restores an etcd member snapshot to an etcd directory
	snapshot status		Gets backend snapshot status of a given file
	make-mirror		Makes a mirror at the destination etcd cluster
	migrate			Migrates keys in a v2 store to a mvcc store
	lock			Acquires a named lock
	elect			Observes and participates in leader election
	auth enable		Enables authentication
	auth disable		Disables authentication
	user add		Adds a new user
	user delete		Deletes a user
	user get		Gets detailed information of a user
	user list		Lists all users
	user passwd		Changes password of user
	user grant-role		Grants a role to a user
	user revoke-role	Revokes a role from a user
	role add		Adds a new role
	role delete		Deletes a role
	role get		Gets detailed information of a role
	role list		Lists all roles
	role grant-permission	Grants a key to a role
	role revoke-permission	Revokes a key from a role
	check perf		Check the performance of the etcd cluster
	help			Help about any command

OPTIONS:
      --cacert=""				verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""					identify secure client using this TLS certificate file
      --command-timeout=5s			timeout for short running command (excluding dial timeout)
      --debug[=false]				enable client-side debug logging
      --dial-timeout=2s				dial timeout for client connections
  -d, --discovery-srv=""			domain name to query for SRV records describing cluster endpoints
      --endpoints=[127.0.0.1:2379]		gRPC endpoints
      --hex[=false]				print byte strings as hex encoded strings
      --insecure-discovery[=true]		accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]	skip server certificate verification
      --insecure-transport[=true]		disable transport security for client connections
      --keepalive-time=2s			keepalive time for client connections
      --keepalive-timeout=6s			keepalive timeout for client connections
      --key=""					identify secure client using this TLS key file
      --user=""					username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"			set the output format (fields, json, protobuf, simple, table)

检索节点信息

etcdctl  member list -w table
+------------------+---------+-----------+------------------------------+------------------------------+
|        ID        | STATUS  |   NAME    |          PEER ADDRS          |         CLIENT ADDRS         |
+------------------+---------+-----------+------------------------------+------------------------------+
| 1c83ad9421d77430 | started | k8s-node3 | https://192.168.174.130:2380 | https://192.168.174.130:2379 |
| 5eec694677c3c515 | started | k8s-node2 | https://192.168.174.129:2380 | https://192.168.174.129:2379 |
| 65f8d952bfce7d85 | started | k8s-node1 | https://192.168.174.128:2380 | https://192.168.174.128:2379 |
+------------------+---------+-----------+------------------------------+------------------------------+

简单命令用用

[root@k8s-node1 ~]# etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.628115ms
[root@k8s-node1 ~]# etcdctl endpoint status
127.0.0.1:2379, 65f8d952bfce7d85, 3.3.7, 23 MB, false, 276, 1555337
[root@k8s-node1 ~]# etcdctl endpoint hashkv
127.0.0.1:2379, 3210071622

检索保存的所有数据目录,所有数据都保存在/registry目录

参数解释,--prefix:默认为true可以看到所有的子目录.--keys-only:默认为true,只显示key,如果设置为false,会显示key的所有值.

以下是检索的所有目录和子目录

etcdctl get /  --prefix --keys-only
/registry/apiregistration.k8s.io/apiservices/v1.

/registry/apiregistration.k8s.io/apiservices/v1.apps

/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.autoscaling

/registry/apiregistration.k8s.io/apiservices/v1.batch

/registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1.storage.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.apiextensions.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.apps

/registry/apiregistration.k8s.io/apiservices/v1beta1.authentication.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.authorization.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.batch

/registry/apiregistration.k8s.io/apiservices/v1beta1.certificates.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.coordination.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.events.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.extensions

/registry/apiregistration.k8s.io/apiservices/v1beta1.networking.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.node.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.policy

/registry/apiregistration.k8s.io/apiservices/v1beta1.rbac.authorization.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.scheduling.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io

/registry/apiregistration.k8s.io/apiservices/v1beta2.apps

/registry/apiregistration.k8s.io/apiservices/v2beta1.autoscaling

/registry/apiregistration.k8s.io/apiservices/v2beta2.autoscaling

/registry/clusterrolebindings/auto-approve-csrs-for-group

/registry/clusterrolebindings/cluster-admin

/registry/clusterrolebindings/kubelet-bootstrap

/registry/clusterrolebindings/node-client-cert-renewal

/registry/clusterrolebindings/node-server-cert-renewal

/registry/clusterrolebindings/system:basic-user

/registry/clusterrolebindings/system:controller:attachdetach-controller

/registry/clusterrolebindings/system:controller:certificate-controller

/registry/clusterrolebindings/system:controller:clusterrole-aggregation-controller

/registry/clusterrolebindings/system:controller:cronjob-controller

/registry/clusterrolebindings/system:controller:daemon-set-controller

/registry/clusterrolebindings/system:controller:deployment-controller

/registry/clusterrolebindings/system:controller:disruption-controller

/registry/clusterrolebindings/system:controller:endpoint-controller

/registry/clusterrolebindings/system:controller:expand-controller

/registry/clusterrolebindings/system:controller:generic-garbage-collector

/registry/clusterrolebindings/system:controller:horizontal-pod-autoscaler

/registry/clusterrolebindings/system:controller:job-controller

/registry/clusterrolebindings/system:controller:namespace-controller

/registry/clusterrolebindings/system:controller:node-controller

/registry/clusterrolebindings/system:controller:persistent-volume-binder

/registry/clusterrolebindings/system:controller:pod-garbage-collector

/registry/clusterrolebindings/system:controller:pv-protection-controller

/registry/clusterrolebindings/system:controller:pvc-protection-controller

/registry/clusterrolebindings/system:controller:replicaset-controller

/registry/clusterrolebindings/system:controller:replication-controller

/registry/clusterrolebindings/system:controller:resourcequota-controller

/registry/clusterrolebindings/system:controller:route-controller

/registry/clusterrolebindings/system:controller:service-account-controller

/registry/clusterrolebindings/system:controller:service-controller

/registry/clusterrolebindings/system:controller:statefulset-controller

/registry/clusterrolebindings/system:controller:ttl-controller

/registry/clusterrolebindings/system:coredns

/registry/clusterrolebindings/system:discovery

/registry/clusterrolebindings/system:kube-controller-manager

/registry/clusterrolebindings/system:kube-dns

/registry/clusterrolebindings/system:kube-scheduler

/registry/clusterrolebindings/system:kubernetes

/registry/clusterrolebindings/system:node

/registry/clusterrolebindings/system:node-proxier

/registry/clusterrolebindings/system:public-info-viewer

/registry/clusterrolebindings/system:user-nonresource-bind

/registry/clusterrolebindings/system:volume-scheduler

/registry/clusterroles/admin

/registry/clusterroles/approve-node-server-renewal-csr

/registry/clusterroles/cluster-admin

/registry/clusterroles/discover_base_url

/registry/clusterroles/edit

/registry/clusterroles/system:aggregate-to-admin

/registry/clusterroles/system:aggregate-to-edit

/registry/clusterroles/system:aggregate-to-view

/registry/clusterroles/system:auth-delegator

/registry/clusterroles/system:basic-user

/registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient

/registry/clusterroles/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient

/registry/clusterroles/system:controller:attachdetach-controller

/registry/clusterroles/system:controller:certificate-controller

/registry/clusterroles/system:controller:clusterrole-aggregation-controller

/registry/clusterroles/system:controller:cronjob-controller

/registry/clusterroles/system:controller:daemon-set-controller

/registry/clusterroles/system:controller:deployment-controller

/registry/clusterroles/system:controller:disruption-controller

/registry/clusterroles/system:controller:endpoint-controller

/registry/clusterroles/system:controller:expand-controller

/registry/clusterroles/system:controller:generic-garbage-collector

/registry/clusterroles/system:controller:horizontal-pod-autoscaler

/registry/clusterroles/system:controller:job-controller

/registry/clusterroles/system:controller:namespace-controller

/registry/clusterroles/system:controller:node-controller

/registry/clusterroles/system:controller:persistent-volume-binder

/registry/clusterroles/system:controller:pod-garbage-collector

/registry/clusterroles/system:controller:pv-protection-controller

/registry/clusterroles/system:controller:pvc-protection-controller

/registry/clusterroles/system:controller:replicaset-controller

/registry/clusterroles/system:controller:replication-controller

/registry/clusterroles/system:controller:resourcequota-controller

/registry/clusterroles/system:controller:route-controller

/registry/clusterroles/system:controller:service-account-controller

/registry/clusterroles/system:controller:service-controller

/registry/clusterroles/system:controller:statefulset-controller

/registry/clusterroles/system:controller:ttl-controller

/registry/clusterroles/system:coredns

/registry/clusterroles/system:csi-external-attacher

/registry/clusterroles/system:csi-external-provisioner

/registry/clusterroles/system:discovery

/registry/clusterroles/system:heapster

/registry/clusterroles/system:kube-aggregator

/registry/clusterroles/system:kube-controller-manager

/registry/clusterroles/system:kube-dns

/registry/clusterroles/system:kube-scheduler

/registry/clusterroles/system:kubelet-api-admin

/registry/clusterroles/system:kubernetes-to-kubelet

/registry/clusterroles/system:node

/registry/clusterroles/system:node-bootstrapper

/registry/clusterroles/system:node-problem-detector

/registry/clusterroles/system:node-proxier

/registry/clusterroles/system:persistent-volume-provisioner

/registry/clusterroles/system:public-info-viewer

/registry/clusterroles/system:volume-scheduler

/registry/clusterroles/view

/registry/configmaps/default/mysql-config

/registry/configmaps/default/mysql-config2

/registry/configmaps/kube-system/coredns

/registry/configmaps/kube-system/extension-apiserver-authentication

/registry/deployments/default/httpd-app

/registry/deployments/default/httpd-pod

/registry/deployments/default/mysql-t

/registry/deployments/default/mysql-test

/registry/deployments/default/wordpress-pod

/registry/deployments/kube-system/coredns

/registry/events/default/busybox.15da92b45565ca8a

/registry/events/default/busybox.15da92b46083229a

/registry/events/default/busybox.15da92b4841c0188

/registry/events/default/httpd.15da92b59a43e346

/registry/events/default/httpd.15da92ba858fde0a

/registry/events/default/httpd.15da92f22acf3927

/registry/leases/kube-node-lease/k8s-node1

/registry/leases/kube-node-lease/k8s-node2

/registry/leases/kube-node-lease/k8s-node3

/registry/masterleases/192.168.174.128

/registry/masterleases/192.168.174.129

/registry/masterleases/192.168.174.130

/registry/minions/k8s-node1

/registry/minions/k8s-node2

/registry/minions/k8s-node3

/registry/namespaces/default

/registry/namespaces/kube-node-lease

/registry/namespaces/kube-public

/registry/namespaces/kube-system

/registry/persistentvolumeclaims/default/mysql-pvc1

/registry/persistentvolumes/mysql-pv1

/registry/pods/default/busybox

/registry/pods/default/httpd

/registry/pods/default/httpd-app-6665fb7898-7x9vd

/registry/pods/default/httpd-app-6665fb7898-8w9k6

/registry/pods/default/httpd-app-6665fb7898-js8vv

/registry/pods/default/httpd-app-6665fb7898-q744z

/registry/pods/default/httpd-app-6665fb7898-xmksq

/registry/pods/default/httpd-pod-586b66458-vs4rm

/registry/pods/default/mysql-t-54666b579c-7m5rv

/registry/pods/default/mysql-test-647b8db96b-qdxw6

/registry/pods/default/wordpress-pod-74c47cd8dd-t4gmr

/registry/pods/kube-system/coredns-5fb99965-brwjq

/registry/pods/kube-system/coredns-5fb99965-svpvn

/registry/priorityclasses/system-cluster-critical

/registry/priorityclasses/system-node-critical

/registry/ranges/serviceips

/registry/ranges/servicenodeports

/registry/replicasets/default/httpd-app-6665fb7898

/registry/replicasets/default/httpd-pod-586b66458

/registry/replicasets/default/httpd-pod-5b8d447886

/registry/replicasets/default/httpd-pod-6bd9576bc9

/registry/replicasets/default/mysql-t-54666b579c

/registry/replicasets/default/mysql-test-647b8db96b

/registry/replicasets/default/wordpress-pod-74c47cd8dd

/registry/replicasets/kube-system/coredns-5fb99965

/registry/rolebindings/default/system:user-nonresource-bind

/registry/rolebindings/kube-public/system:controller:bootstrap-signer

/registry/rolebindings/kube-system/system::extension-apiserver-authentication-reader

/registry/rolebindings/kube-system/system::leader-locking-kube-controller-manager

/registry/rolebindings/kube-system/system::leader-locking-kube-scheduler

/registry/rolebindings/kube-system/system:controller:bootstrap-signer

/registry/rolebindings/kube-system/system:controller:cloud-provider

/registry/rolebindings/kube-system/system:controller:token-cleaner

/registry/roles/kube-public/system:controller:bootstrap-signer

/registry/roles/kube-system/extension-apiserver-authentication-reader

/registry/roles/kube-system/system::leader-locking-kube-controller-manager

/registry/roles/kube-system/system::leader-locking-kube-scheduler

/registry/roles/kube-system/system:controller:bootstrap-signer

/registry/roles/kube-system/system:controller:cloud-provider

/registry/roles/kube-system/system:controller:token-cleaner

/registry/secrets/default/default-token-fwtch

/registry/secrets/default/mysecret

/registry/secrets/default/mysecret2

/registry/secrets/kube-node-lease/default-token-76lpr

/registry/secrets/kube-public/default-token-6fvpw

/registry/secrets/kube-system/attachdetach-controller-token-7hc8p

/registry/secrets/kube-system/bootstrap-signer-token-7xshb

/registry/secrets/kube-system/certificate-controller-token-dwqx4

/registry/secrets/kube-system/clusterrole-aggregation-controller-token-gr77d

/registry/secrets/kube-system/coredns-token-52bg6

/registry/secrets/kube-system/cronjob-controller-token-2xkff

/registry/secrets/kube-system/daemon-set-controller-token-pcdln

/registry/secrets/kube-system/default-token-82pm9

/registry/secrets/kube-system/deployment-controller-token-2t9xj

/registry/secrets/kube-system/disruption-controller-token-q982v

/registry/secrets/kube-system/endpoint-controller-token-28zb9

/registry/secrets/kube-system/expand-controller-token-f9ht9

/registry/secrets/kube-system/generic-garbage-collector-token-6llv2

/registry/secrets/kube-system/horizontal-pod-autoscaler-token-6kdkn

/registry/secrets/kube-system/job-controller-token-7dgtm

/registry/secrets/kube-system/namespace-controller-token-8w94q

/registry/secrets/kube-system/node-controller-token-x4q8s

/registry/secrets/kube-system/persistent-volume-binder-token-pknv8

/registry/secrets/kube-system/pod-garbage-collector-token-gr2c4

/registry/secrets/kube-system/pv-protection-controller-token-ctcrj

/registry/secrets/kube-system/pvc-protection-controller-token-7kc5j

/registry/secrets/kube-system/replicaset-controller-token-5z6sw

/registry/secrets/kube-system/replication-controller-token-jgk96

/registry/secrets/kube-system/resourcequota-controller-token-zbx8g

/registry/secrets/kube-system/service-account-controller-token-cnsn6

/registry/secrets/kube-system/service-controller-token-gv47w

/registry/secrets/kube-system/statefulset-controller-token-np2qv

/registry/secrets/kube-system/token-cleaner-token-mtpht

/registry/secrets/kube-system/ttl-controller-token-lr8nz

/registry/serviceaccounts/default/default

/registry/serviceaccounts/kube-node-lease/default

/registry/serviceaccounts/kube-public/default

/registry/serviceaccounts/kube-system/attachdetach-controller

/registry/serviceaccounts/kube-system/bootstrap-signer

/registry/serviceaccounts/kube-system/certificate-controller

/registry/serviceaccounts/kube-system/clusterrole-aggregation-controller

/registry/serviceaccounts/kube-system/coredns

/registry/serviceaccounts/kube-system/cronjob-controller

/registry/serviceaccounts/kube-system/daemon-set-controller

/registry/serviceaccounts/kube-system/default

/registry/serviceaccounts/kube-system/deployment-controller

/registry/serviceaccounts/kube-system/disruption-controller

/registry/serviceaccounts/kube-system/endpoint-controller

/registry/serviceaccounts/kube-system/expand-controller

/registry/serviceaccounts/kube-system/generic-garbage-collector

/registry/serviceaccounts/kube-system/horizontal-pod-autoscaler

/registry/serviceaccounts/kube-system/job-controller

/registry/serviceaccounts/kube-system/namespace-controller

/registry/serviceaccounts/kube-system/node-controller

/registry/serviceaccounts/kube-system/persistent-volume-binder

/registry/serviceaccounts/kube-system/pod-garbage-collector

/registry/serviceaccounts/kube-system/pv-protection-controller

/registry/serviceaccounts/kube-system/pvc-protection-controller

/registry/serviceaccounts/kube-system/replicaset-controller

/registry/serviceaccounts/kube-system/replication-controller

/registry/serviceaccounts/kube-system/resourcequota-controller

/registry/serviceaccounts/kube-system/service-account-controller

/registry/serviceaccounts/kube-system/service-controller

/registry/serviceaccounts/kube-system/statefulset-controller

/registry/serviceaccounts/kube-system/token-cleaner

/registry/serviceaccounts/kube-system/ttl-controller

/registry/services/endpoints/default/httpd-svc

/registry/services/endpoints/default/kubernetes

/registry/services/endpoints/default/mysql-t

/registry/services/endpoints/default/mysql-test

/registry/services/endpoints/default/wordpress

/registry/services/endpoints/kube-system/kube-controller-manager

/registry/services/endpoints/kube-system/kube-dns

/registry/services/endpoints/kube-system/kube-scheduler

/registry/services/specs/default/httpd-svc

/registry/services/specs/default/kubernetes

/registry/services/specs/default/mysql-t

/registry/services/specs/default/mysql-test

/registry/services/specs/default/wordpress

/registry/services/specs/kube-system/kube-dns

检索pod信息

先用kubectl命令看看多少pod

kubectl get pod  --all-namespaces
NAMESPACE     NAME                             READY   STATUS             RESTARTS   AGE
default       busybox                          1/1     Running            23         5d22h
default       httpd                            0/1     CrashLoopBackOff   131        23h
default       httpd-app-6665fb7898-7x9vd       1/1     Running            4          7d3h
default       httpd-app-6665fb7898-8w9k6       1/1     Running            4          7d3h
default       httpd-app-6665fb7898-js8vv       1/1     Running            4          7d3h
default       httpd-app-6665fb7898-q744z       1/1     Running            4          7d3h
default       httpd-app-6665fb7898-xmksq       1/1     Running            4          7d3h
default       httpd-pod-586b66458-vs4rm        1/1     Running            3          6d4h
default       mysql-t-54666b579c-7m5rv         1/1     Running            2          3d23h
default       mysql-test-647b8db96b-qdxw6      1/1     Running            1          21h
default       wordpress-pod-74c47cd8dd-t4gmr   1/1     Running            3          5d22h
kube-system   coredns-5fb99965-brwjq           1/1     Running            12         20d
kube-system   coredns-5fb99965-svpvn           1/1     Running            12         20d

etcd里查找pod信息,见下

 etcdctl get /registry/pods  --prefix --keys-only
/registry/pods/default/busybox

/registry/pods/default/httpd

/registry/pods/default/httpd-app-6665fb7898-7x9vd

/registry/pods/default/httpd-app-6665fb7898-8w9k6

/registry/pods/default/httpd-app-6665fb7898-js8vv

/registry/pods/default/httpd-app-6665fb7898-q744z

/registry/pods/default/httpd-app-6665fb7898-xmksq

/registry/pods/default/httpd-pod-586b66458-vs4rm

/registry/pods/default/mysql-t-54666b579c-7m5rv

/registry/pods/default/mysql-test-647b8db96b-qdxw6

/registry/pods/default/wordpress-pod-74c47cd8dd-t4gmr

/registry/pods/kube-system/coredns-5fb99965-brwjq

/registry/pods/kube-system/coredns-5fb99965-svpvn

检索单个pod详细信息,输出格式有四种,但是protobuf和simple会乱码,这里使用json格式,见下

etcdctl get /registry/pods/default/busybox  -w json
{"header":{"cluster_id":8418215620332445483,"member_id":7347861741483490693,"revision":596755,"raft_term":276},"kvs":[{"key":"L3JlZ2lzdHJ5L3BvZHMvZGVmYXVsdC9idXN5Ym94","create_revision":448381,"mod_revision":594625,"version":52,"value":"azhzAAoJCgJ2MRIDUG9kEqIMCowECgdidXN5Ym94EgAaB2RlZmF1bHQiACokYzZmZjg2YmItMGM5ZC00NTg0LWFhOTQtYjczN2Q5ZTFhMWI4MgA4AEIICMSA1O4FEABivQMKMGt1YmVjdGwua3ViZXJuZXRlcy5pby9sYXN0LWFwcGxpZWQtY29uZmlndXJhdGlvbhKIA3siYXBpVmVyc2lvbiI6InYxIiwia2luZCI6IlBvZCIsIm1ldGFkYXRhIjp7ImFubm90YXRpb25zIjp7fSwibmFtZSI6ImJ1c3lib3giLCJuYW1lc3BhY2UiOiJkZWZhdWx0In0sInNwZWMiOnsiY29udGFpbmVycyI6W3siY29tbWFuZCI6WyJzbGVlcCIsIjM2MDAiXSwiaW1hZ2UiOiJidXN5Ym94OjEuMjguNCIsImltYWdlUHVsbFBvbGljeSI6IklmTm90UHJlc2VudCIsIm5hbWUiOiJidXN5Ym94Iiwidm9sdW1lTW91bnRzIjpbeyJtb3VudFBhdGgiOiIvZXRjL3Rlc3QiLCJuYW1lIjoidGVzdCIsInJlYWRPbmx5Ijp0cnVlfV19XSwicmVzdGFydFBvbGljeSI6IkFsd2F5cyIsInZvbHVtZXMiOlt7Im5hbWUiOiJ0ZXN0Iiwic2VjcmV0Ijp7InNlY3JldE5hbWUiOiJteXNlY3JldDIifX1dfX0KegAS5wMKGAoEdGVzdBIQMg4KCW15c2VjcmV0MhikAwoxChNkZWZhdWx0LXRva2VuLWZ3dGNoEhoyGAoTZGVmYXVsdC10b2tlbi1md3RjaBikAxLDAQoHYnVzeWJveBIOYnVzeWJveDoxLjI4LjQaBXNsZWVwGgQzNjAwKgBCAEoXCgR0ZXN0EAEaCS9ldGMvdGVzdCIAMgBKSgoTZGVmYXVsdC10b2tlbi1md3RjaBABGi0vdmFyL3J1bi9zZWNyZXRzL2t1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQiADIAahQvZGV2L3Rlcm1pbmF0aW9uLWxvZ3IMSWZOb3RQcmVzZW50gAEAiAEAkAEAogEERmlsZRoGQWx3YXlzIB4yDENsdXN0ZXJGaXJzdEIHZGVmYXVsdEoHZGVmYXVsdFIJazhzLW5vZGUxWABgAGgAcgCCAQCKAQCaARFkZWZhdWx0LXNjaGVkdWxlcrIBNgocbm9kZS5rdWJlcm5ldGVzLmlvL25vdC1yZWFkeRIGRXhpc3RzGgAiCU5vRXhlY3V0ZSisArIBOAoebm9kZS5rdWJlcm5ldGVzLmlvL3VucmVhY2hhYmxlEgZFeGlzdHMaACIJTm9FeGVjdXRlKKwCwgEAyAEA8AEBGqYECgdSdW5uaW5nEiMKC0luaXRpYWxpemVkEgRUcnVlGgAiCAjEgNTuBRAAKgAyABIdCgVSZWFkeRIEVHJ1ZRoAIggIia3z7gUQACoAMgASJwoPQ29udGFpbmVyc1JlYWR5EgRUcnVlGgAiCAiJrfPuBRAAKgAyABIkCgxQb2RTY2hlZHVsZWQSBFRydWUaACIICMSA1O4FEAAqADIAGgAiACoPMTkyLjE2OC4xNzQuMTI4MgsxNzIuMzAuNzcuMzoICMSA1O4FEABCzQIKB2J1c3lib3gSDBIKCggIiK3z7gUQABpyGnAIABAAGglDb21wbGV0ZWQiACoICPeQ8+4FEAAyCAiHrfPuBRAAOklkb2NrZXI6Ly9jNTQ3YmFlZWZkZTFmNDg5ZGE4NTc0YTFkYjhlZWZhYWJhYTViNzFhYWJiYjNmOGYyOWIzNjk1YmU2ZGQ4OGJlIAEoGDIOYnVzeWJveDoxLjI4LjQ6YWRvY2tlci1wdWxsYWJsZTovL2J1c3lib3hAc2hhMjU2OjE0MWMyNTNiYzRjM2ZkMGEyMDFkMzJkYzFmNDkzYmNmM2ZmZjAwM2I2ZGY0MTZkZWE0ZjQxMDQ2ZTBmMzdkNDdCSWRvY2tlcjovLzhkZGU1NDc2ODAxMWY1YWI4ZmIwYWVmNzJjNjg0YWEwNTQyNDNlZTU1NTdiN2EyMzQ3OWNiMDQwY2I1ODA3YzFKCkJlc3RFZmZvcnRaABoAIgA="}],"count":1}

也可以用curl命令来获取这个信息,见下

可以通过命令来获取pod的etcd存储路径selflink

kubectl get pod busybox -o yaml |grep self
  selfLink: /api/v1/namespaces/default/pods/busybox

curl https://192.168.174.127:8443/api/v1/namespaces/default/pods/busybox  --cacert /etc/kubernetes/cert/ca.pem --cert /etc/kubernetes/cert/kubernetes.pem --key /etc/kubernetes/cert/kubernetes-key.pem
{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "busybox",
    "namespace": "default",
    "selfLink": "/api/v1/namespaces/default/pods/busybox",
    "uid": "c6ff86bb-0c9d-4584-aa94-b737d9e1a1b8",
    "resourceVersion": "594625",
    "creationTimestamp": "2019-11-20T08:58:44Z",
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"busybox\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"command\":[\"sleep\",\"3600\"],\"image\":\"busybox:1.28.4\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"busybox\",\"volumeMounts\":[{\"mountPath\":\"/etc/test\",\"name\":\"test\",\"readOnly\":true}]}],\"restartPolicy\":\"Always\",\"volumes\":[{\"name\":\"test\",\"secret\":{\"secretName\":\"mysecret2\"}}]}}\n"
    }
  },
  "spec": {
    "volumes": [
      {
        "name": "test",
        "secret": {
          "secretName": "mysecret2",
          "defaultMode": 420
        }
      },
      {
        "name": "default-token-fwtch",
        "secret": {
          "secretName": "default-token-fwtch",
          "defaultMode": 420
        }
      }
    ],
    "containers": [
      {
        "name": "busybox",
        "image": "busybox:1.28.4",
        "command": [
          "sleep",
          "3600"
        ],
        "resources": {
          
        },
        "volumeMounts": [
          {
            "name": "test",
            "readOnly": true,
            "mountPath": "/etc/test"
          },
          {
            "name": "default-token-fwtch",
            "readOnly": true,
            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
          }
        ],
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "IfNotPresent"
      }
    ],
    "restartPolicy": "Always",
    "terminationGracePeriodSeconds": 30,
    "dnsPolicy": "ClusterFirst",
    "serviceAccountName": "default",
    "serviceAccount": "default",
    "nodeName": "k8s-node1",
    "securityContext": {
      
    },
    "schedulerName": "default-scheduler",
    "tolerations": [
      {
        "key": "node.kubernetes.io/not-ready",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      },
      {
        "key": "node.kubernetes.io/unreachable",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      }
    ],
    "priority": 0,
    "enableServiceLinks": true
  },
  "status": {
    "phase": "Running",
    "conditions": [
      {
        "type": "Initialized",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-11-20T08:58:44Z"
      },
      {
        "type": "Ready",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-11-26T07:38:49Z"
      },
      {
        "type": "ContainersReady",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-11-26T07:38:49Z"
      },
      {
        "type": "PodScheduled",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-11-20T08:58:44Z"
      }
    ],
    "hostIP": "192.168.174.128",
    "podIP": "172.30.77.3",
    "startTime": "2019-11-20T08:58:44Z",
    "containerStatuses": [
      {
        "name": "busybox",
        "state": {
          "running": {
            "startedAt": "2019-11-26T07:38:48Z"
          }
        },
        "lastState": {
          "terminated": {
            "exitCode": 0,
            "reason": "Completed",
            "startedAt": "2019-11-26T06:38:47Z",
            "finishedAt": "2019-11-26T07:38:47Z",
            "containerID": "docker://c547baeefde1f489da8574a1db8eefaabaa5b71aabbb3f8f29b3695be6dd88be"
          }
        },
        "ready": true,
        "restartCount": 24,
        "image": "busybox:1.28.4",
        "imageID": "docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47",
        "containerID": "docker://8dde54768011f5ab8fb0aef72c684aa054243ee5557b7a23479cb040cb5807c1"
      }
    ],
    "qosClass": "BestEffort"
  }

通过etcd删除pod,删除下面两个pod

kubectl get pod
NAME                             READY   STATUS             RESTARTS   AGE
httpd-app-6665fb7898-7x9vd       1/1     Running            4          7d4h
httpd-app-6665fb7898-8w9k6       1/1     Running            4          7d4h
httpd-app-6665fb7898-js8vv       1/1     Running            4          7d4h
httpd-app-6665fb7898-q744z       1/1     Running            4          7d4h
httpd-app-6665fb7898-xmksq       1/1     Running            4          7d4h
httpd-pod-586b66458-vs4rm        1/1     Running            3          6d5h

检索deployment

etcdctl get /registry/deployment --keys-only --prefix
/registry/deployments/default/httpd-app

/registry/deployments/default/httpd-pod

删除

 etcdctl del /registry/deployments/default/httpd-app
1
 etcdctl del /registry/deployments/default/httpd-pod
1

删除后检索,已经删除了

kubectl get deployment
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
mysql-t         1/1     1            1           4d
mysql-test      1/1     1            1           22h
wordpress-pod   1/1     1            1           5d23h

kubectl get pod
NAME                             READY   STATUS             RESTARTS   AGE
busybox                          1/1     Running            24         5d23h
httpd                            0/1     CrashLoopBackOff   151        24h
mysql-t-54666b579c-7m5rv         1/1     Running            2          4d
mysql-test-647b8db96b-qdxw6      1/1     Running            1          22h
wordpress-pod-74c47cd8dd-t4gmr   1/1     Running            3          5d23h

4.etcd数据备份

最好做个脚本每天备份

备份

etcdctl --endpoints 127.0.0.1:2379 snapshot save snashot.db
Snapshot saved at snashot.db

 ll -h
total 22M
-rw-r--r-- 1 root root 22M Nov 26 03:25 snashot.db

删除一个节点的数据再恢复

执行命令:systemctl stop etcd

执行命令:rm -rf /var/lib/etcd/

恢复

etcdctl --name=k8s-node1 --endpoints="https://192.168.174.128:2379" --cacert=/etc/kubernetes/cert/ca.pem --key=/etc/etcd/cert/etcd-key.pem --cert=/etc/etcd/cert/etcd.pem --initial-cluster-token=etcd-cluster-1 --initial-advertise-peer-urls=https://192.168.174.128:2380 --initial-cluster=k8s-node1=https://192.168.174.128:2380,k8s-node2=https://192.168.174.129:2380,k8s-node3=https://192.168.174.130:2380 --data-dir=/var/lib/etcd snapshot restore snashot.db
2019-11-26 03:36:36.038458 I | mvcc: restore compact to 598557
2019-11-26 03:36:36.046152 I | etcdserver/membership: added member 189141ea20f3026c [https://192.168.174.128:2380] to cluster 5a726f403538a406
2019-11-26 03:36:36.046181 I | etcdserver/membership: added member 2a5472cf06cd4b08 [https://192.168.174.129:2380] to cluster 5a726f403538a406
2019-11-26 03:36:36.046192 I | etcdserver/membership: added member d5b1c5fde22a3133 [https://192.168.174.130:2380] to cluster 5a726f403538a406

重启服务

systemctl daemon-reload && systemctl restart etcd
Job for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details.

服务起不来,报错

Nov 26 03:39:23 k8s-node1 etcd: etcd Version: 3.3.7
Nov 26 03:39:23 k8s-node1 etcd: Git SHA: 56536de55
Nov 26 03:39:23 k8s-node1 etcd: Go Version: go1.9.6
Nov 26 03:39:23 k8s-node1 etcd: Go OS/Arch: linux/amd64
Nov 26 03:39:23 k8s-node1 etcd: setting maximum number of CPUs to 1, total number of available CPUs is 1
Nov 26 03:39:23 k8s-node1 etcd: error listing data dir: /var/lib/etcd
Nov 26 03:39:23 k8s-node1 systemd: etcd.service: main process exited, code=exited, status=1/FAILURE
Nov 26 03:39:23 k8s-node1 systemd: Failed to start Etcd Server.
Nov 26 03:39:23 k8s-node1 systemd: Unit etcd.service entered failed state.
Nov 26 03:39:23 k8s-node1 systemd: etcd.service failed.

检查权限

修改数据目录权限,默认是root:root
chown -R k8s:k8s /var/lib/etcd

再重新启动,还是报错

systemctl status etcd -l
● etcd.service - Etcd Server
   Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: activating (start) since Tue 2019-11-26 03:42:09 EST; 37s ago
     Docs: https://github.com/coreos
 Main PID: 46204 (etcd)
    Tasks: 6
   Memory: 29.2M
   CGroup: /system.slice/etcd.service
           └─46204 /opt/k8s/bin/etcd --data-dir=/var/lib/etcd --name=k8s-node1 --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem --trusted-ca-file=/etc/kubernetes/cert/ca.pem --peer-cert-file=/etc/etcd/cert/etcd.pem --peer-key-file=/etc/etcd/cert/etcd-key.pem --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem --peer-client-cert-auth --client-cert-auth --listen-peer-urls=https://192.168.174.128:2380 --initial-advertise-peer-urls=https://192.168.174.128:2380 --listen-client-urls=https://192.168.174.128:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.174.128:2379 --initial-cluster-token=etcd-cluster-0 --initial-cluster=k8s-node1=https://192.168.174.128:2380,k8s-node2=https://192.168.174.129:2380,k8s-node3=https://192.168.174.130:2380 --initial-cluster-state=new

Nov 26 03:42:47 k8s-node1 etcd[46204]: request sent was ignored (cluster ID mismatch: peer[2a5472cf06cd4b08]=74d382814c8f5f2b, local=5a726f403538a406)
Nov 26 03:42:47 k8s-node1 etcd[46204]: request sent was ignored (cluster ID mismatch: peer[d5b1c5fde22a3133]=74d382814c8f5f2b, local=5a726f403538a406)
Nov 26 03:42:47 k8s-node1 etcd[46204]: request cluster ID mismatch (got 74d382814c8f5f2b want 5a726f403538a406)
Nov 26 03:42:47 k8s-node1 etcd[46204]: request cluster ID mismatch (got 74d382814c8f5f2b want 5a726f403538a406)
Nov 26 03:42:47 k8s-node1 etcd[46204]: request sent was ignored (cluster ID mismatch: peer[d5b1c5fde22a3133]=74d382814c8f5f2b, local=5a726f403538a406)
Nov 26 03:42:47 k8s-node1 etcd[46204]: request sent was ignored (cluster ID mismatch: peer[2a5472cf06cd4b08]=74d382814c8f5f2b, local=5a726f403538a406)
Nov 26 03:42:47 k8s-node1 etcd[46204]: request cluster ID mismatch (got 74d382814c8f5f2b want 5a726f403538a406)
Nov 26 03:42:47 k8s-node1 etcd[46204]: request cluster ID mismatch (got 74d382814c8f5f2b want 5a726f403538a406)
Nov 26 03:42:47 k8s-node1 etcd[46204]: request cluster ID mismatch (got 74d382814c8f5f2b want 5a726f403538a406)
Nov 26 03:42:47 k8s-node1 etcd[46204]: request cluster ID mismatch (got 74d382814c8f5f2b want 5a726f403538a406)

很无奈的报错,只能删除所有节点的etcd存储目录数据,然后恢复

所有节点做下面操作

执行命令:systemctl stop etcd
执行命令:rm -rf /var/lib/etcd/

恢复

[root@k8s-node1 etcd_db]# etcdctl --name=k8s-node1 --endpoints="https://192.168.174.128:2379" --cacert=/etc/kubernetes/cert/ca.pem --key=/etc/etcd/cert/etcd-key.pem --cert=/etc/etcd/cert/etcd.pem --initial-cluster-token=etcd-cluster-1 --initial-advertise-peer-urls=https://192.168.174.128:2380 --initial-cluster=k8s-node1=https://192.168.174.128:2380,k8s-node2=https://192.168.174.129:2380,k8s-node3=https://192.168.174.130:2380 --data-dir=/var/lib/etcd snapshot restore snashot.db

[root@k8s-node2 ~]# etcdctl --name=k8s-node2 --endpoints="https://192.168.174.129:2379" --cacert=/etc/kubernetes/cert/ca.pem --key=/etc/etcd/cert/etcd-key.pem --cert=/etc/etcd/cert/etcd.pem --initial-cluster-token=etcd-cluster-1 --initial-advertise-peer-urls=https://192.168.174.129:2380 --initial-cluster=k8s-node1=https://192.168.174.128:2380,k8s-node2=https://192.168.174.129:2380,k8s-node3=https://192.168.174.130:2380 --data-dir=/var/lib/etcd snapshot restore snashot.db


[root@k8s-node3 ~]# etcdctl --name=k8s-node3 --endpoints="https://192.168.174.130:2379" --cacert=/etc/kubernetes/cert/ca.pem --key=/etc/etcd/cert/etcd-key.pem --cert=/etc/etcd/cert/etcd.pem --initial-cluster-token=etcd-cluster-1 --initial-advertise-peer-urls=https://192.168.174.130:2380 --initial-cluster=k8s-node1=https://192.168.174.128:2380,k8s-node2=https://192.168.174.129:2380,k8s-node3=https://192.168.174.130:2380 --data-dir=/var/lib/etcd snapshot restore snashot.db

修改权限

chown -R k8s:k8s /var/lib/etcd

重启服务

systemctl daemon-reload && systemctl restart etcd

服务起来了

[root@k8s-node1 etcd_db]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-11-26 03:53:26 EST; 25s ago
     Docs: https://github.com/coreos
 Main PID: 50373 (etcd)
    Tasks: 7
   Memory: 38.6M
   CGroup: /system.slice/etcd.service
           └─50373 /opt/k8s/bin/etcd --data-dir=/var/lib/etcd --name=k8s-node1 --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem --trusted-ca...

Nov 26 03:53:32 k8s-node1 etcd[50373]: read-only range request "key:\"/registry/deployments\" range_end:\"/registry/deploymentt\" count_only:true " took too...to execute
Nov 26 03:53:32 k8s-node1 etcd[50373]: setting up the initial cluster version to 3.3
Nov 26 03:53:32 k8s-node1 etcd[50373]: set the initial cluster version to 3.3
Nov 26 03:53:32 k8s-node1 etcd[50373]: enabled capabilities for version 3.3
Nov 26 03:53:33 k8s-node1 etcd[50373]: read-only range request "key:\"/registry/validatingwebhookconfigurations/\" range_end:\"/registry/validatingwebhookco...to execute
Nov 26 03:53:33 k8s-node1 etcd[50373]: read-only range request "key:\"/registry/controllerrevisions/\" range_end:\"/registry/controllerrevisions0\" limit:10...to execute
Nov 26 03:53:33 k8s-node1 etcd[50373]: read-only range request "key:\"/registry/deployments/\" range_end:\"/registry/deployments0\" limit:10000 " took too l...to execute
Nov 26 03:53:33 k8s-node1 etcd[50373]: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/\" range_end:\"/registry/apiregistration....to execute
Nov 26 03:53:33 k8s-node1 etcd[50373]: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/\" range_end:\"/registry/apiregistration....to execute
Nov 26 03:53:33 k8s-node1 etcd[50373]: read-only range request "key:\"/registry/apiregistration.k8s.io/apiservices/\" range_end:\"/registry/apiregistration....to execute
Hint: Some lines were ellipsized, use -l to show in full.
[root@k8s-node1 etcd_db]# etcdctl member list
189141ea20f3026c, started, k8s-node1, https://192.168.174.128:2380, https://192.168.174.128:2379
2a5472cf06cd4b08, started, k8s-node2, https://192.168.174.129:2380, https://192.168.174.129:2379
d5b1c5fde22a3133, started, k8s-node3, https://192.168.174.130:2380, https://192.168.174.130:2379

[root@k8s-node1 etcd_db]# kubectl get all --all-namespaces
NAMESPACE     NAME                                 READY   STATUS             RESTARTS   AGE
default       pod/busybox                          1/1     Running            24         5d23h
default       pod/httpd                            0/1     CrashLoopBackOff   163        25h
default       pod/mysql-t-54666b579c-7m5rv         1/1     Running            2          4d1h
default       pod/mysql-test-647b8db96b-qdxw6      1/1     Running            1          23h
default       pod/wordpress-pod-74c47cd8dd-t4gmr   1/1     Running            3          6d
kube-system   pod/coredns-5fb99965-brwjq           1/1     Running            12         20d
kube-system   pod/coredns-5fb99965-svpvn           1/1     Running            12         20d


NAMESPACE     NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default       service/httpd-svc    NodePort    10.254.125.1     <none>        80:8400/TCP              7d5h
default       service/kubernetes   ClusterIP   10.254.0.1       <none>        443/TCP                  21d
default       service/mysql-t      ClusterIP   10.254.177.63    <none>        3306/TCP                 4d1h
default       service/mysql-test   ClusterIP   10.254.177.188   <none>        3306/TCP                 23h
default       service/wordpress    NodePort    10.254.208.45    <none>        8080:8800/TCP            6d
kube-system   service/kube-dns     ClusterIP   10.254.0.2       <none>        53/UDP,53/TCP,9153/TCP   20d


NAMESPACE     NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
default       deployment.apps/mysql-t         1/1     1            1           4d1h
default       deployment.apps/mysql-test      1/1     1            1           23h
default       deployment.apps/wordpress-pod   1/1     1            1           6d
kube-system   deployment.apps/coredns         2/2     2            2           20d

NAMESPACE     NAME                                       DESIRED   CURRENT   READY   AGE
default       replicaset.apps/mysql-t-54666b579c         1         1         1       4d1h
default       replicaset.apps/mysql-test-647b8db96b      1         1         1       23h
default       replicaset.apps/wordpress-pod-74c47cd8dd   1         1         1       6d
kube-system   replicaset.apps/coredns-5fb99965           2         2         2       20d

flanneld因为是v2的api,数据丢失,没有恢复

systemctl status flanneld
● flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/etc/systemd/system/flanneld.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-11-25 20:38:06 EST; 7h ago
 Main PID: 1593 (flanneld)
    Tasks: 7
   Memory: 28.1M
   CGroup: /system.slice/flanneld.service
           └─1593 /opt/k8s/bin/flanneld -etcd-cafile=/etc/kubernetes/cert/ca.pem -etcd-certfile=/etc/flanneld/cert/flanneld.pem -etcd-keyfile=/etc/flanneld/cert/flann...

Nov 26 03:53:20 k8s-node1 flanneld[1593]: ; error #2: dial tcp 192.168.174.129:2379: getsockopt: connection refused
Nov 26 03:53:20 k8s-node1 flanneld[1593]: E1126 03:53:20.760062    1593 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured; er...on refused
Nov 26 03:53:20 k8s-node1 flanneld[1593]: ; error #1: dial tcp 192.168.174.128:2379: getsockopt: connection refused
Nov 26 03:53:20 k8s-node1 flanneld[1593]: ; error #2: dial tcp 192.168.174.129:2379: getsockopt: connection refused
Nov 26 03:53:21 k8s-node1 flanneld[1593]: E1126 03:53:21.754201    1593 watch.go:171] Subnet watch failed: client: etcd cluster is unavailable or misconfigu...on refused
Nov 26 03:53:21 k8s-node1 flanneld[1593]: ; error #1: dial tcp 192.168.174.128:2379: getsockopt: connection refused
Nov 26 03:53:21 k8s-node1 flanneld[1593]: ; error #2: dial tcp 192.168.174.129:2379: getsockopt: connection refused
Nov 26 03:53:21 k8s-node1 flanneld[1593]: E1126 03:53:21.764048    1593 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured; er...on refused
Nov 26 03:53:21 k8s-node1 flanneld[1593]: ; error #1: dial tcp 192.168.174.128:2379: getsockopt: connection refused
Nov 26 03:53:21 k8s-node1 flanneld[1593]: ; error #2: dial tcp 192.168.174.129:2379: getsockopt: connection refused

export ETCDCTL_API=2
etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem ls /kubernetes/network/subnets
Error:  100: Key not found (/kubernetes) [7]

重新生成flanneld的网络信息

[root@k8s-node1 etcd_db]# source /opt/k8s/bin/environment.sh
[root@k8s-node1 etcd_db]# echo ${ETCD_ENDPOINTS}
https://192.168.174.128:2379,https://192.168.174.129:2379,https://192.168.174.130:2379
[root@k8s-node1 etcd_db]# export ETCDCTL_API=2
[root@k8s-node1 etcd_db]# etcdctl --endpoints=${ETCD_ENDPOINTS} --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/flanneld/cert/flanneld.pem --key-file=/etc/flanneld/cert/flanneld-key.pem set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'","SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
{"Network":"172.30.0.0/16","SubnetLen": 24, "Backend": {"Type": "vxlan"}}

重启flanneld服务,恢复正常

systemctl daemon-reload && systemctl restart flanneld

 systemctl status flanneld
● flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/etc/systemd/system/flanneld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-11-26 04:02:21 EST; 5s ago
  Process: 54301 ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
 Main PID: 54294 (flanneld)
    Tasks: 7
   Memory: 18.9M
   CGroup: /system.slice/flanneld.service
           └─54294 /opt/k8s/bin/flanneld -etcd-cafile=/etc/kubernetes/cert/ca.pem -etcd-certfile=/etc/flanneld/cert/flanneld.pem -etcd-keyfile=/etc/flanneld/cert/flan...

Nov 26 04:02:20 k8s-node1 flanneld[54294]: I1126 04:02:20.936155   54294 main.go:238] Installing signal handlers
Nov 26 04:02:20 k8s-node1 flanneld[54294]: I1126 04:02:20.950943   54294 main.go:353] Found network config - Backend type: vxlan
Nov 26 04:02:20 k8s-node1 flanneld[54294]: I1126 04:02:20.950982   54294 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
Nov 26 04:02:20 k8s-node1 flanneld[54294]: I1126 04:02:20.957841   54294 local_manager.go:201] Found previously leased subnet (172.30.77.0/24), reusing
Nov 26 04:02:20 k8s-node1 flanneld[54294]: I1126 04:02:20.971954   54294 local_manager.go:220] Allocated lease (172.30.77.0/24) to current node (192.168.174.128)
Nov 26 04:02:20 k8s-node1 flanneld[54294]: I1126 04:02:20.972219   54294 main.go:300] Wrote subnet file to /run/flannel/subnet.env
Nov 26 04:02:20 k8s-node1 flanneld[54294]: I1126 04:02:20.972227   54294 main.go:304] Running backend.
Nov 26 04:02:20 k8s-node1 flanneld[54294]: I1126 04:02:20.979256   54294 vxlan_network.go:60] watching for new subnet leases
Nov 26 04:02:20 k8s-node1 flanneld[54294]: I1126 04:02:20.996038   54294 main.go:396] Waiting for 22h59m59.969177096s to renew lease
Nov 26 04:02:21 k8s-node1 systemd[1]: Started Flanneld overlay address etcd agent.

etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem get /kubernetes/network/config
{"Network":"172.30.0.0/16","SubnetLen": 24, "Backend": {"Type": "vxlan"}}

 etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/etcd/cert/etcd.pem --key-file=/etc/etcd/cert/etcd-key.pem ls /kubernetes/network/subnets
/kubernetes/network/subnets/172.30.49.0-24
/kubernetes/network/subnets/172.30.77.0-24
/kubernetes/network/subnets/172.30.75.0-24

kubectl cluster-info
Kubernetes master is running at https://192.168.174.127:8443
CoreDNS is running at https://192.168.174.127:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.