statefulset svc statefulset svc全称地址

StatefulSet

• StatefulSet（有状态集）常用于部署有状态的且需要有序启动的应用程序。

StatefulSet的基本概念

• StatefulSet主要用于管理有状态应用程序的工作负载API对象.
• 比如在生产环境中，可以部署ElasticSearch集群、MongoDB集群或者需要持久化的RabbitMQ集群、Redis集群、Kafka集群和ZooKeeper集群等。
• 而StatefulSet创建的Pod一般使用Headless Service（无头服务）进行通信，和普通的Service的区别在于Headless Service没有ClusterIP，它使用的是Endpoint进行互相通信，
• Headless一般的格式为

statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local。

# 说明：
- serviceName为Headless Service的名字。
- 0..N-1为Pod所在的序号，从0开始到N-1。
- statefulSetName为StatefulSet的名字。
- namespace为服务所在的命名空间。
- .cluster.local为Cluster Domain（集群域）。

• 比如，一个Redis主从架构，Slave连接Master主机配置就可以使用不会更改的Master的Headless Service，例如Redis从节点（Slave）配置文件如下：

port 6379
slaveofredis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local 6379
tcp-backlog 511
timeout 0
tcp-keepalive 0

……

• 其中，redis-sentinel-master-ss-0.redis-sentinel-master-ss.public-service.svc.cluster.local是Redis Master的Headless Service。

定义一个简单的StatefulSet

apiVersion: v1
kind: Service
metadata:
  name: nginx-sts
  labels:
    app: nginx-sts
spec:
  ports:
  - port: 80
    name: nginx-sts
  clusterIP: None
  selector:
    app: nginx-sts
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx-sts
spec:
  serviceName: "nginx-sts"
  replicas: 2
  selector:
    matchLabels:
      app: nginx-sts
  template:
    metadata:
      labels:
        app: nginx-sts
    spec:
      containers:
      - name: nginx-sts
        image: nginx:1.15.4
        ports:
        - containerPort: 80
          name: nginx-sts

• kind: Service:

• 定义了一个名字为Nginx的Headless Service，创建的Service格式为nginx-0.nginx.default.svc.cluster.local，其他的类似，因为没有指定Namespace（命名空间），所以默认部署在default。

• kind: StatefulSet:

• 定义了一个名字为web的StatefulSet，replicas表示部署Pod的副本数，本实例为2。

在StatefulSet中必须设置Pod选择器（.spec.selector）用来匹配其标签（.spec.template.metadata.labels）。在1.8版本之前，如果未配置该字段（.spec.selector），将被设置为默认值，在1.8版本之后，如果未指定匹配Pod Selector，则会导致StatefulSet创建错误。

当StatefulSet控制器创建Pod时，它会添加一个标签statefulset.kubernetes.io/pod-name，该标签的值为Pod的名称，用于匹配Service。

查看创建的service和pod

# service CLUSTER-IP为None
[root@k8s-master-01 k8s_test]# kubectl  get svc  nginx-sts
NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
nginx-sts   ClusterIP   None         <none>        80/TCP    2m53s

# pod
[root@k8s-master-01 k8s_test]# kubectl  get pod  -owide
NAME          READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
nginx-sts-0   1/1     Running   0          8m22s   10.96.44.201   k8s-node-02   <none>           <none>
nginx-sts-1   1/1     Running   0          7m34s   10.96.89.135   k8s-node-03   <none>           <none>

扩容statefulset

[root@k8s-master-01 k8s_test]# kubectl scale --replicas=3 sts nginx-sts 
statefulset.apps/nginx-sts scaled

[root@k8s-master-01 k8s_test]# kubectl  get pod  -owide
NAME          READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
nginx-sts-0   1/1     Running   0          11m   10.96.44.201    k8s-node-02   <none>           <none>
nginx-sts-1   1/1     Running   0          10m   10.96.89.135    k8s-node-03   <none>           <none>
nginx-sts-2   1/1     Running   0          62s   10.96.154.201   k8s-node-01   <none>           <none>

测试statefulset

• 创建一个busybox pod用于测试

[root@k8s-master-01 k8s_test]# cat busybox_test.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
 
[root@k8s-master-01 k8s_test]# kubectl apply -f busybox_test.yaml 
pod/busybox created

[root@k8s-master-01 k8s_test]# kubectl  get po busybox 
NAME      READY   STATUS    RESTARTS   AGE
busybox   1/1     Running   0          29s

• 进入busybox 进行测试

[root@k8s-master-01 k8s_test]# kubectl  exec -it busybox  -- sh

# 根据podname.service可以解析出pod的ip
/ # nslookup nginx-sts-0.nginx-sts
Server:    10.244.0.10
Address 1: 10.244.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx-sts-0.nginx-sts
Address 1: 10.96.44.201 nginx-sts-0.nginx-sts.default.svc.cluster.local


/ # nslookup nginx-sts-1.nginx-sts
Server:    10.244.0.10
Address 1: 10.244.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx-sts-1.nginx-sts
Address 1: 10.96.89.135 nginx-sts-1.nginx-sts.default.svc.cluster.local

StatefulSet 扩缩容

• 当生成statefulset的pod时, 会按顺序一个生成好之后再生成下一个,
• 比如当把3个副本的nginx-sts扩容到6个副本, 会依次创建nginx-sts-3, nginx-sts-4,nginx-sts-5
• 当创建过程中前面某个pod被删除, 比如nginx-sts-0被删除了, k8s会将nginx-sts-0创建好之后再继续创建后面的pod

[root@k8s-master-01 k8s_test]# kubectl scale --replicas=6 sts nginx-sts 
statefulset.apps/nginx-sts scaled

[root@k8s-master-01 k8s_test]# kubectl  delete po nginx-sts-0
pod "nginx-sts-0" deleted

# 使用命令实时查看pod状态, 另开一个窗口扩容前执行
[root@k8s-master-01 ~]# kubectl  get po -l app=nginx-sts -w
NAME          READY   STATUS    RESTARTS   AGE
nginx-sts-0   1/1     Running   0          53s
nginx-sts-1   1/1     Running   0          55m
nginx-sts-2   1/1     Running   0          3m34s
nginx-sts-3   0/1     Pending   0          0s
nginx-sts-3   0/1     Pending   0          0s
nginx-sts-3   0/1     ContainerCreating   0          0s
nginx-sts-3   0/1     ContainerCreating   0          1s
nginx-sts-3   1/1     Running             0          1s  # nginx-sts-3 创建完成
nginx-sts-4   0/1     Pending             0          0s  # 开始创建nginx-sts-4
nginx-sts-4   0/1     Pending             0          0s
nginx-sts-4   0/1     ContainerCreating   0          0s
nginx-sts-0   1/1     Terminating         0          67s # 删除nginx-sts-0
nginx-sts-4   0/1     ContainerCreating   0          1s
nginx-sts-0   1/1     Terminating         0          67s
nginx-sts-0   0/1     Terminating         0          67s
nginx-sts-0   0/1     Terminating         0          67s
nginx-sts-0   0/1     Terminating         0          67s # nginx-sts-0删除完成
nginx-sts-0   0/1     Pending             0          0s  # 开始创建nginx-sts-0
nginx-sts-0   0/1     Pending             0          0s
nginx-sts-0   0/1     ContainerCreating   0          0s
nginx-sts-0   0/1     ContainerCreating   0          1s
nginx-sts-4   1/1     Running             0          2s # nginx-sts-4创建完成
nginx-sts-0   0/1     ContainerCreating   0          1s 
nginx-sts-0   1/1     Running             0          1s # nginx-sts-0创建完成
nginx-sts-5   0/1     Pending             0          0s
nginx-sts-5   0/1     Pending             0          0s
nginx-sts-5   0/1     ContainerCreating   0          0s
nginx-sts-5   0/1     ContainerCreating   0          1s
nginx-sts-5   1/1     Running             0          2s

• 当缩容删除pod时,会从最后一个序号的pod开始, 一个一个的删除,
• 如果删除过程中前面某个pod被删除了,
• 比如nginx-sts-0被删除了 k8s会继续当前正在删除的pod和nginx-sts-0的删除操作,
• 等当前删除的pod和nginx-sts-0完全删除, 再nginx-sts-0创建好之后再继续删除操作

# 缩容
[root@k8s-master-01 k8s_test]# kubectl scale --replicas=2 sts nginx-sts 
statefulset.apps/nginx-sts scaled

# 删除nginx-sts0
[root@k8s-master-01 k8s_test]# kubectl  delete po nginx-sts-0
pod "nginx-sts-0" deleted

# 查看pod状态,另开一个窗口缩容前执行
[root@k8s-master-01 ~]# kubectl  get po -l app=nginx-sts -w
NAME          READY   STATUS    RESTARTS   AGE
nginx-sts-0   1/1     Running   0          37m
nginx-sts-1   1/1     Running   0          37m
nginx-sts-2   1/1     Running   0          14s
nginx-sts-3   1/1     Running   0          12s
nginx-sts-4   1/1     Running   0          11s
nginx-sts-4   1/1     Terminating   0          49s
nginx-sts-4   1/1     Terminating   0          49s  # nginx-sts-4 还没删除完
nginx-sts-0   1/1     Terminating   0          38m  # 执行删除nginx-sts-0命令
nginx-sts-0   1/1     Terminating   0          38m
nginx-sts-4   0/1     Terminating   0          50s  
nginx-sts-4   0/1     Terminating   0          50s
nginx-sts-4   0/1     Terminating   0          50s  # nginx-sts-4 删除完成
nginx-sts-0   0/1     Terminating   0          38m
nginx-sts-0   0/1     Terminating   0          38m
nginx-sts-0   0/1     Terminating   0          38m  # nginx-sts-0 删除完成
nginx-sts-0   0/1     Pending       0          0s   # 开始创建nginx-sts-0
nginx-sts-0   0/1     Pending       0          0s
nginx-sts-0   0/1     ContainerCreating   0          0s
nginx-sts-0   0/1     ContainerCreating   0          1s
nginx-sts-0   1/1     Running             0          1s  # nginx-sts-0创建完成
nginx-sts-3   1/1     Terminating         0          52s  # 继续删除nginx-sts-3
nginx-sts-3   1/1     Terminating         0          53s
nginx-sts-3   0/1     Terminating         0          53s
nginx-sts-3   0/1     Terminating         0          53s
nginx-sts-3   0/1     Terminating         0          53s
nginx-sts-2   1/1     Terminating         0          55s  # 继续删除nginx-sts-2
nginx-sts-2   1/1     Terminating         0          55s
nginx-sts-2   0/1     Terminating         0          56s
nginx-sts-2   0/1     Terminating         0          56s
nginx-sts-2   0/1     Terminating         0          56s

StatefulSet 更新策略

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"nginx-sts","namespace":"default"},"spec":{"replicas":2,"selector":{"matchLabels":{"app":"nginx-sts"}},"serviceName":"nginx-sts","template":{"metadata":{"labels":{"app":"nginx-sts"}},"spec":{"containers":[{"image":"nginx:1.15.4","name":"nginx-sts","ports":[{"containerPort":80,"name":"nginx-sts"}]}]}}}}
  creationTimestamp: "2022-05-24T14:44:47Z"
  generation: 10
  name: nginx-sts
  namespace: default
  resourceVersion: "150803"
  uid: fb68f471-2b7d-4449-b8d7-b21614a1ef41
spec:
  podManagementPolicy: OrderedReady
  replicas: 6
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-sts
  serviceName: nginx-sts
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-sts
    spec:
      containers:
      - image: nginx:1.15.4
        imagePullPolicy: IfNotPresent
        name: nginx-sts
        ports:
        - containerPort: 80
          name: nginx-sts
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:    	                # 更新策略
    type: RollingUpdate                 # RollingUpdate|OnDelete
    rollingUpdate:                      # RollingUpdate选项
      partition: 0                      # 当type为RollingUpdate时保留不更新的副本数, 可实现灰度发布,分段更新

RollingUpdate

• RollingUpdate为默认更新策略, pod序号从大到小依次更新
• 比如一个statefulset: nginx-sts,它有3个副本, 会从nginx-sts-2开始, 然后nginx-sts-1, 最后更新nginx-sts-0

[root@k8s-master-01 ~]# kubectl set image sts nginx-sts nginx-sts=nginx:1.15.4  --record
statefulset.apps/nginx-sts image updated

# 实时查看pod状态
[root@k8s-master-01 ~]# kubectl  get po -l app=nginx-sts -w
NAME          READY   STATUS    RESTARTS   AGE
nginx-sts-0   1/1     Running   0          2m2s
nginx-sts-1   1/1     Running   0          2m22s
nginx-sts-2   1/1     Running   0          3m3s
nginx-sts-2   1/1     Terminating   0          3m9s        # 开始删除nginx-sts-2
nginx-sts-2   1/1     Terminating   0          3m9s
nginx-sts-2   0/1     Terminating   0          3m10s
nginx-sts-2   0/1     Terminating   0          3m10s
nginx-sts-2   0/1     Terminating   0          3m10s
nginx-sts-2   0/1     Pending       0          0s          # 开始创建nginx-sts-2
nginx-sts-2   0/1     Pending       0          0s
nginx-sts-2   0/1     ContainerCreating   0          0s
nginx-sts-2   0/1     ContainerCreating   0          0s
nginx-sts-2   1/1     Running             0          1s    # 完成创建nginx-sts-2
nginx-sts-1   1/1     Terminating         0          2m30s # 开始删除nginx-sts-1
nginx-sts-1   1/1     Terminating         0          2m30s 
nginx-sts-1   0/1     Terminating         0          2m31s
nginx-sts-1   0/1     Terminating         0          2m31s
nginx-sts-1   0/1     Terminating         0          2m31s
nginx-sts-1   0/1     Pending             0          0s    # 开始创建nginx-sts-1
nginx-sts-1   0/1     Pending             0          0s
nginx-sts-1   0/1     ContainerCreating   0          0s
nginx-sts-1   0/1     ContainerCreating   0          0s
nginx-sts-1   1/1     Running             0          1s    # 完成创建nginx-sts-1
nginx-sts-0   1/1     Terminating         0          2m12s # 开始删除nginx-sts-0
nginx-sts-0   1/1     Terminating         0          2m13s
nginx-sts-0   0/1     Terminating         0          2m13s
nginx-sts-0   0/1     Terminating         0          2m13s
nginx-sts-0   0/1     Terminating         0          2m13s
nginx-sts-0   0/1     Pending             0          0s    # 开始创建nginx-sts-0
nginx-sts-0   0/1     Pending             0          0s
nginx-sts-0   0/1     ContainerCreating   0          0s
nginx-sts-0   0/1     ContainerCreating   0          0s
nginx-sts-0   0/1     ContainerCreating   0          0s
nginx-sts-0   1/1     Running             0          1s    # 完成创建nginx-sts-

• partition

当type为RollingUpdate时保留不更新的副本数, 可实现灰度发布,分段更新

# 用命令将nginx-sts的partition修改为2
[root@k8s-master-01 ~]# kubectl  edit sts nginx-sts 
statefulset.apps/nginx-sts edited

# 如下
  updateStrategy:
    rollingUpdate:
      partition: 2
    type: RollingUpdate

# 修改sts的镜像版本
kubectl set image sts nginx-sts nginx-sts=nginx:1.15.1  --record

# 实时查看pod状态
[root@k8s-master-01 ~]# kubectl  get po -l app=nginx-sts -w
NAME          READY   STATUS    RESTARTS   AGE
nginx-sts-0   1/1     Running   0          14m
nginx-sts-1   1/1     Running   0          14m
nginx-sts-2   1/1     Running   0          6s
nginx-sts-2   1/1     Terminating   0          11s  # 可以看到只有nginx-sts-2被删除重建了, 
nginx-sts-2   1/1     Terminating   0          11s
nginx-sts-2   0/1     Terminating   0          11s
nginx-sts-2   0/1     Terminating   0          11s
nginx-sts-2   0/1     Terminating   0          11s
nginx-sts-2   0/1     Pending       0          0s
nginx-sts-2   0/1     Pending       0          0s
nginx-sts-2   0/1     ContainerCreating   0          0s
nginx-sts-2   0/1     ContainerCreating   0          1s
nginx-sts-2   1/1     Running             0          1s

# 用命令将nginx-sts的partition修改为1
[root@k8s-master-01 ~]# kubectl  edit sts nginx-sts 
statefulset.apps/nginx-sts edited

# 然后nginx-sts-1也自动删除重建了
nginx-sts-1   1/1     Terminating         0          16m
nginx-sts-1   1/1     Terminating         0          16m
nginx-sts-1   0/1     Terminating         0          16m
nginx-sts-1   0/1     Terminating         0          16m
nginx-sts-1   0/1     Terminating         0          16m
nginx-sts-1   0/1     Pending             0          0s
nginx-sts-1   0/1     Pending             0          0s
nginx-sts-1   0/1     ContainerCreating   0          0s
nginx-sts-1   0/1     ContainerCreating   0          0s
nginx-sts-1   1/1     Running             0          1s

OnDelete

当updateStrategy.type为OnDelete时, 对statefulset做的修改不会立即更新, 只有当pod被手动删除,或其他原因重启时才会触发更新

# 用命令将nginx-sts的updateStrategy.type修改为OnDelete
[root@k8s-master-01 ~]# kubectl  edit sts nginx-sts 
statefulset.apps/nginx-sts edited

# 如下
  updateStrategy:
    type: OnDelete

# 修改nginx-sts的镜像版本
kubectl set image sts nginx-sts nginx-sts=nginx:1.15.3  --record

# 此时pod并没有自动更新
[root@k8s-master-01 ~]# kubectl  get po -l app=nginx-sts -w
NAME          READY   STATUS    RESTARTS   AGE
nginx-sts-0   1/1     Running   0          4m35s
nginx-sts-1   1/1     Running   0          5m25s
nginx-sts-2   1/1     Running   0          7m38s

# 删除nginx-sts-0
[root@k8s-master-01 ~]# kubectl  get po -l app=nginx-sts -w
NAME          READY   STATUS    RESTARTS   AGE
nginx-sts-0   1/1     Running   0          4m35s
nginx-sts-1   1/1     Running   0          5m25s
nginx-sts-2   1/1     Running   0          7m38s
nginx-sts-0   1/1     Terminating   0          5m58s
nginx-sts-0   1/1     Terminating   0          5m58s
nginx-sts-0   0/1     Terminating   0          5m58s
nginx-sts-0   0/1     Terminating   0          5m58s
nginx-sts-0   0/1     Terminating   0          5m58s
nginx-sts-0   0/1     Pending       0          0s
nginx-sts-0   0/1     Pending       0          0s
nginx-sts-0   0/1     ContainerCreating   0          0s
nginx-sts-0   0/1     ContainerCreating   0          1s
nginx-sts-0   1/1     Running             0          1s   # nginx-sts-0重建完成

# 检查三个pod的镜像版本 只有nginx-sts-0镜像版本更新了.
[root@k8s-master-01 ~]# kubectl  get pod nginx-sts-0 -oyaml | grep 'image:'
  - image: nginx:1.15.3
    image: docker.io/library/nginx:1.15.3
[root@k8s-master-01 ~]# kubectl  get pod nginx-sts-1 -oyaml | grep 'image:'
  - image: nginx:1.15.1
    image: docker.io/library/nginx:1.15.1
[root@k8s-master-01 ~]# kubectl  get pod nginx-sts-2 -oyaml | grep 'image:'
  - image: nginx:1.15.1
    image: docker.io/library/nginx:1.15.1

statefulset级联删除与非级联删除

级联删除

默认就是级联删除,在删除statefulset时, 连同这个statefulset的pod一起删除

# 删除statefulset
[root@k8s-master-01 ~]# kubectl  delete sts nginx-sts 
statefulset.apps "nginx-sts" deleted

# pod也一起被自动删除了
[root@k8s-master-01 ~]# kubectl  get po -l app=nginx-sts 
No resources found in default namespace.

非级联删除

当删除statefulset时填加参数–cascade=orphan, 此时不会立即删除该statefulset相关的pod, 这些pod在下次被手动删除时就会被彻底删除, 不再重建

# 创建nginx-sts
[root@k8s-master-01 k8s_test]# kubectl  create -f nginx-sts_test.yaml 
statefulset.apps/nginx-sts created
Error from server (AlreadyExists): error when creating "nginx-sts_test.yaml": services "nginx-sts" already exists

# 非级联删除nginx-sts
[root@k8s-master-01 k8s_test]# kubectl  delete sts nginx-sts --cascade=orphan
statefulset.apps "nginx-sts" deleted

# pod并没有被删除
[root@k8s-master-01 k8s_test]# kubectl  get pod 
NAME          READY   STATUS    RESTARTS       AGE
busybox       1/1     Running   10 (39m ago)   23h
nginx-sts-0   1/1     Running   0              5m21s
nginx-sts-1   1/1     Running   0              5m19s

# 手动删除这两个孤儿pod
[root@k8s-master-01 k8s_test]# kubectl  delete po nginx-sts-0 nginx-sts-1
pod "nginx-sts-0" deleted
pod "nginx-sts-1" deleted

# 孤儿pod没有被重建
[root@k8s-master-01 k8s_test]# kubectl  get pod 
NAME      READY   STATUS    RESTARTS       AGE
busybox   1/1     Running   10 (41m ago)   23h