Prometheus 介绍 prometheus operator

目录标题

• 一、Prometheus-Operator介绍
• 二、Prometheus-Operator安装
• 三、通过Ingress访问组件
• 四、添加监控对象
• 五、自定义报警规则

• 1. 指定Alertmanager地址
• 2. 添加报警规则

• 六、配置微信方式报警
• 七、自动发现配置
• 八、使用NFS持久化数据

一、Prometheus-Operator介绍

Prometheus Operator 是了为简化在 Kubernetes 上部署、管理和运行 Prometheus 和 Alertmanager 集群而设计的，它为监控 Kubernetes 资源和 Prometheus 实例的管理提供了简单的定义，比如自带了一些报警规则和展示看板。

Prometheus Operator 架构图

Prometheus Operator 组件：

• Operator：控制器，根据自定义资源来部署和管理 Prometheus Server；
• Prometheus Server： 根据自定义资源 Prometheus 类型中定义的内容而部署的 Prometheus Server 集群，这些自定义资源可以看作是用来管理 Prometheus Server 集群的 StatefulSets 资源；
• Prometheus：声明 Prometheus 资源对象期望的状态，Operator 确保这个资源对象运行时一直与定义保持一致；
• ServiceMonitor：声明指定监控的服务， 也就是exporter 的抽象，通过 Labels 来选取对应的Service Endpoint，让 Prometheus Server 通过选取的 Service 来获取 Metrics 信息。
• Service：需要监控的服务，简单的说就是 Prometheus 监控的对象。

二、Prometheus-Operator安装

可以使用Helm方式安装，这里选择的是手动安装

下载Prometheus-Operator项目到本地服务器

$ git clone https://github.com/coreos/kube-prometheus.git
$ cd manifests

安装setup目录下的CRD和Operator对象

$ kubectl  apply  -f setup/
namespace/monitoring created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created

创建manifests目录下的各类资源

$ kubectl  apply -f  .
alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
.............
$ kubectl  get pods -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    1/2     Running   0          2d17h
alertmanager-main-1                    1/2     Running   0          2d17h
alertmanager-main-2                    1/2     Running   0          2d17h
grafana-86445dccbb-m7kzg               1/1     Running   0          2d17h
kube-state-metrics-5b67d79459-zf27k    3/3     Running   0          2d17h
node-exporter-blx8m                    2/2     Running   0          2d17h
node-exporter-zpns2                    2/2     Running   0          2d17h
node-exporter-zrd6g                    2/2     Running   0          2d17h
prometheus-adapter-66b855f564-mf9mc    1/1     Running   0          2d17h
prometheus-k8s-0                       3/3     Running   1          2d17h
prometheus-k8s-1                       3/3     Running   1          2d17h
prometheus-operator-78fcb48ccf-sgklz   2/2     Running   0          2d17h

三、通过Ingress访问组件

由于这些资源的默认Service为ClusterIP，集群外部无法访问，我们可以通过使用kubectl edit xxx命令将Service类型修改为NodePort方式来提供外部访问。

这里使用Ingress 分别为Prometheus，Alertmanager，Grafana创建域名。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: monitoring
  name: prometheus-ingress
spec:
  rules:
  - host: k8s.grafana.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 3000
  - host: k8s.prometheus.com
    http:
      paths:
      - backend:
          serviceName: prometheus-k8s
          servicePort: 9090
  - host: k8s.alertmanager.com
    http:
      paths:
      - backend:
          serviceName: alertmanager-main
          servicePort: 9093

创建Ingress对象

$ kubectl  create -f ingress.yaml 
ingress.extensions/prometheus-ingress created
$ kubectl  get ingress -A
NAMESPACE    NAME                 CLASS    HOSTS                                                     ADDRESS   PORTS     AGE
default      my-nginx             <none>   nginx.ingress.com                                                   80, 443   2d21h                        
monitoring   prometheus-ingress   <none>   k8s.grafana.com,k8s.prometheus.com,k8s.alertmanager.com             80        4s

可以看到，已经分别创建了相应的域名，我们在本地的hosts文件添加Ingress主机的IP地址解析即可通过域名访问了。

四、添加监控对象

在上面Kube-proemtheus默认监控了一些系统的组件，我们还需要根据实际的业务需求去添加自定义的组件监控，添加自定义监控对象步骤如下：

1. 建立一个 ServiceMonitor 对象，用于 Prometheus 添加监控项
2. 为 ServiceMonitor 对象关联 metrics 数据接口的一个 Service 对象
3. 确保 Service 对象可以正确获取到 metrics 数据

比如对etcd服务进行监控，先创建ServiceMonitor对象

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: etcd-k8s
 namespace: monitoring
 labels:
   k8s-app: etcd-k8s
spec:
 jobLabel: k8s-app
 endpoints:
 - port: port
   interval: 15s
 selector:
   matchLabels:
     k8s-app: etcd
 namespaceSelector:
   matchNames:
   - kube-system

定义为：匹配 kube-system 这个命名空间下面的具有k8s-app=etcd 这个 label 标签的 Service，其中jobLabel 表示用于检索 job 任务名称的标签。

创建这个serviceMonitor对象

$ kubectl apply -f prometheus-serviceMonitorEtcd.yaml
servicemonitor.monitoring.coreos.com "etcd-k8s" created

然后再创建一个Etcd的Service 对象

apiVersion: v1
kind: Service
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
spec:
  type: ClusterIP
  clusterIP: None  # 一定要设置 clusterIP:None
  ports:
  - name: port
    port: 2381
---
apiVersion: v1
kind: Endpoints
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
subsets:
- addresses:
  - ip: 192.168.16.173  # 指定etcd节点地址，如果是集群则继续向下添加
    nodeName: etc-master
  ports:
  - name: port
    port: 2381

上面文件定义为：将后端的Etcd服务通过Endpoints添加到集群，然后为其创建Service对象

创建这个Service对象

$ kubectl apply -f etcd-service.yaml
service/etcd-k8s configured
endpoints/etcd-k8s configured
$ kubectl get svc -n kube-system -l k8s-app=etcd
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
etcd-k8s   ClusterIP   None         <none>        2381/TCP   1d

创建完成后去prommetheus查看targes

可以看到2381 端口链接被拒绝，这是因为在创建etcd服务时metrics接口设置为- --listen-metrics-urls=http://127.0.0.1:2381，我们只需要修改 /etc/kubernetes/manifest/ 目录下面的 etcd.yaml 文件中将上面的listen-metrics-urls更改成节点 IP 即可：

- --listen-metrics-urls=http://192.168.16.173:2381

注意：我的单节点etcd开启metrics端口后会不断异常重启，推测是压力扛不住？

修改后etcd会自动重启，然后在去Prometheus查看是否正常

然后在Grafana导入编号为 3070 的 dashboard，就可以获取到 etcd 的监控图表：(grafana默认口令为admin/admin)

五、自定义报警规则

1. 指定Alertmanager地址

在之前使用部署Prometheus时，我们只需要修改Prometheus下的Prometheus.yaml 配置文件来指定Alertmanager地址即可。现在通过Operator方式部署的Prometheus如何指定呢？我们可以先去Prometheus的web页面上查看Configuration的配置信息

可以看到上面 alertmanagers 的配置是通过 role 为 endpoints 的 kubernetes 的自动发现机制获取的，匹配的是服务名为 alertmanager-main，端口名为 web 的 Service 服务，所以Prometheus就这样指定了Alertmanager的地址。

2. 添加报警规则

在上面的配置中可以看到规则文件的路径为： /etc/prometheus/rules/prometheus-k8s-rulefiles-0/*.yaml，我们可以进入Prometheus的容器中查看：

$ kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring
Defaulting container name to prometheus.
Use 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod.
/prometheus $ ls /etc/prometheus/rules/prometheus-k8s-rulefiles-0/
monitoring-prometheus-k8s-rules.yaml
/prometheus $ cat /etc/prometheus/rules/prometheus-k8s-rulefiles-0/monitoring-pr
ometheus-k8s-rules.yaml
groups:
- name: k8s.rules
  rules:
  - expr: |
      sum(rate(container_cpu_usage_seconds_total{job="kubelet", image!="", container_name!=""}[5m])) by (namespace)
    record: namespace:container_cpu_usage_seconds_total:sum_rate
......

而这个文件实际上就是我们之前创建的一个 PrometheusRule 文件包含的内容：

$ cat manifests/prometheus-rules.yaml |  head -10
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: prometheus-k8s-rules
  namespace: monitoring
spec:
  groups:

这个文件中有非常重要的一个属性ruleSelector，用来匹配 rule 规则的过滤器，要求匹配具有 prometheus=k8s 和 role=alert-rules 标签的 PrometheusRule 资源对象。

ruleSelector:
  matchLabels:
    prometheus: k8s
    role: alert-rules

所以我们想要添加规则时，只需要创建一个具有prometheus=k8s 和 role=alert-rules标签的 PrometheusRule 对象就可以了，比如我们对刚才添加的etcd服务编写一条是否可用的报警规则。

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: etcd-rules
  namespace: monitoring
spec:
  groups:
  - name: etcd
    rules:
    - alert: EtcdClusterUnavailable
      annotations:
        summary: etcd cluster small
        description: If one more etcd peer goes down the cluster will be unavailable
      expr: |
        count(up{job="etcd"} == 0) > (count(up{job="etcd"}) / 2 - 1)
      for: 3m
      labels:
        severity: critical

创建这个PrometheusRule对象，然后查看Prometheus容器是否有这个报警规则文件

$ kubectl  create -f etcd-rules.yaml 
prometheusrule.monitoring.coreos.com/etcd-rules created
[root@k8s-master01 manifests]# kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring
$ kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
Defaulting container name to prometheus.
Use 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod.
/prometheus $ ls  /etc/prometheus/rules/prometheus-k8s-rulefiles-0/
monitoring-etcd-rules.yaml            monitoring-prometheus-k8s-rules.yaml

然后再去prometheus的web页面查看下

六、配置微信方式报警

现在报警规则有了，但是报警渠道还没有配置，所以下面我们来修改下Alertmanager的配置，首先我们可以去 Alertmanager 的页面上 status 路径下面查看 AlertManager 的配置信息:

这些配置的来由也是我们之前创建的alertmanger-secret文件。

$ cat  manifests/alertmanager-secret.yaml 
apiVersion: v1
data: {}
kind: Secret
metadata:
  name: alertmanager-main
  namespace: monitoring
stringData:
  alertmanager.yaml: |-
    "global":
      "resolve_timeout": "5m"
    "inhibit_rules":
    - "equal":
      - "namespace"
      - "alertname"
      "source_match":
        "severity": "critical"
      "target_match_re":
        "severity": "warning|info"
    - "equal":
      - "namespace"
      - "alertname"
      "source_match":
        "severity": "warning"
      "target_match_re":
        "severity": "info"
    "receivers":
    - "name": "Default"
    - "name": "Watchdog"
    - "name": "Critical"
    "route":
      "group_by":
      - "namespace"
      "group_interval": "5m"
      "group_wait": "30s"
      "receiver": "Default"
      "repeat_interval": "12h"
      "routes":
      - "match":
          "alertname": "Watchdog"
        "receiver": "Watchdog"
      - "match":
          "severity": "critical"
        "receiver": "Critical"
type: Opaque

然后我们就可以通过修改这个yaml文件来指定我们的报警渠道了，比如我们这里将critical级别的报警发送到微信。

apiVersion: v1
data: {}
kind: Secret
metadata:
  name: alertmanager-main
  namespace: monitoring
stringData:
  alertmanager.yaml: |-
    "global":
      "resolve_timeout": "5m"
    "inhibit_rules":
    - "equal":
      - "namespace"
      - "alertname"
      "source_match":
        "severity": "critical"
      "target_match_re":
        "severity": "warning|info"
    - "equal":
      - "namespace"
      - "alertname"
      "source_match":
        "severity": "warning"
      "target_match_re":
        "severity": "info"
    "receivers":
    - "name": "Default"
    - "name": "Watchdog"
    - "name": "Critical"
      "wechat_configs":  #添加微信的认证
       - "corp_id": 'ww314010b4720f24'
         "to_party": '1'
         "agent_id": '1000002'
         "api_secret": '9nmYzEg8X860ZBIoOkToCbh_oNc'
         "send_resolved": true
    "route":
      "group_by":
      - "namespace"
      "group_interval": "5m"
      "group_wait": "30s"
      "receiver": "Default"
      "repeat_interval": "12h"
      "routes":
      - "match":
          "alertname": "Watchdog"
        "receiver": "Watchdog"
      - "match":
          "severity": "critical"
        "receiver": "Critical"
type: Opaque

然后强制更新alertmanager-secret对象

$ kubectl  delete -f alertmanager-secret.yaml 
ksecret "alertmanager-main" deleted
$ kubectl   apply -f alertmanager-secret.yaml 
secret/alertmanager-main created

然后查看alertmanger的web页面中的配置信息是否加载

如果有critical级别的报警，微信就会收到报警信息

七、自动发现配置

当集群中的Service和Pod越来越多时，我们再手动的为每一个服务创建相应的ServiceMonitor就很麻烦了，所以为解决这个问题，Prometheus Operator 为我们提供了一个额外的抓取配置的来解决这个问题，我们可以通过添加额外的配置来进行服务发现进行自动监控。

新建prometheus-additional.yaml

- job_name: 'kubernetes-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

通过这个文件创建一个对应的 Secret 对象：

$ kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring
secret "additional-configs" created

然后我们需要在声明 prometheus 的资源对象文件中通过additionalScrapeConfigs 属性添加上这个额外的配置：

$ cat prometheus-prometheus.yaml
.................
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.20.0
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml

添加完成后，更新 prometheus 这个 CRD 资源对象

$ kubectl apply -f prometheus-prometheus.yaml
prometheus.monitoring.coreos.com/k8s configured

然后我们就可以在Prometheus的可视化页面查看是否加载了该配置

但是在 targets 页面下面并没有对应的监控任务，查看 Prometheus 的 Pod 日志：

$ kubectl logs -f prometheus-k8s-0 prometheus -n monitoring
.............
level=error ts=2020-10-27T06:01:30.129Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2020-10-27T06:01:39.194Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"

这个报错的原因是因为Prometheus绑定了一个名为 prometheus-k8s 的 ServiceAccount 对象，而这个ServiceAccount账户绑定的是一个名为 prometheus-k8s 的 ClusterRole集群角色，查看这个ClusterRole的权限

$ cat prometheus-clusterRole.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

可以看到这个clusterRole没有对 Service 或者 Pod 的 list 权限，添加上对应权限应该就可以了。

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

重建Prometheus的所有对象

$ cd manifests
$ mkdir prometheus
$ mv prometheus-* prometheus
$ cd prometheus
$ kubectl delete -f .
$ kubectl apply  -f .

重建完成后就可以看到 targets 页面下面有 kubernetes-endpoints 这个监控任务了：

可以看到，抓取到了kube-dns这个Service，这是因为Service 中含有 prometheus.io/scrape=true这个 annotation，可以查看下kube-dns 的service信息

$  kubectl   describe svc kube-dns -n kube-system
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       prometheus.io/port: 9153
                   prometheus.io/scrape: true 
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         10.244.0.2:53,10.244.0.3:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         10.244.0.2:53,10.244.0.3:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         10.244.0.2:9153,10.244.0.3:9153
Session Affinity:  None
Events:            <none>

所以我们在创建Service的时候就要添加 prometheus.io/scrape=true 这个annotations，才可以被Prometheus的服务发现抓取到，添加方式如下：

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  creationTimestamp: "2020-10-26T08:39:47Z"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
............

注意： 虽然应用添加了这个annotations信息，Prometheus也可以抓取到这个目标，但前提是这个应用必须提供了metics接口来暴露指标信息，可以通过 prometheus.io/port: "9153"这个annotations来指定metics接口，否则prometheus采集不到metics信息，则会认为这个服务是DOWN状态。

八、使用NFS持久化数据

在上面我们重建了Prometheus的Pod，查看时会发现之前的数据都丢失了，这是因为我们通过 prometheus 这个 CRD 创建的 Prometheus 并没有做数据的持久化，

$ kubectl get pod prometheus-k8s-0 -n monitoring -o yaml
......
    volumeMounts:
    - mountPath: /etc/prometheus/config_out
      name: config-out
      readOnly: true
    - mountPath: /prometheus
      name: prometheus-k8s-db
......
  volumes:
......
  - emptyDir: {}
    name: prometheus-k8s-db
......

可以看到 Prometheus 的数据目录 /prometheus 实际上是通过emptyDir进行挂载的，而emptyDir的设计就是应用删除后，数据也会删除，所以我们需要对Prometheus的数据进行持久化。

Prometheus是通过 Statefulset 控制器进行部署的，所以我们这里通过 storageclass 来做数据持久化，这里我们选择之前搭建的NFS StorageClass作为数据持久化。

在Prometheus的CRD对象中添加storage属性：

$ cat  prometheus-prometheus.yaml 
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  storage:         #持久化存储
    volumeClaimTemplate:
      spec:
        storageClassName: nfs-data-db
        resources:
          requests:
            storage: 10Gi
  image: quay.io/prometheus/prometheus:v2.20.0
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.20.0
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml

更新prometheus CRD 资源

$ kubectl  apply -f prometheus-prometheus.yaml 
prometheus.monitoring.coreos.com/k8s configured

查看PVC状态

$ kubectl   get pvc -n monitoring
NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prometheus-k8s-db-prometheus-k8s-0   Bound    pvc-8e73e8b2-1e98-452c-8aaa-a9ba694fe234   10Gi       RWO            nfs-data-db    2m
prometheus-k8s-db-prometheus-k8s-1   Bound    pvc-fe817cdd-812f-489e-b82c-d5de7f0dbf93   10Gi       RWO            nfs-data-db    2m