grafana集成es grafana 集群

转载

IT智行领袖 2024-04-03 00:05:21

文章标签 prometheus grafana kubernetes Powered by 金山文档数据 文章分类 架构后端开发

在k8s集群中使用node-exporter、prometheus、grafana对集群进行监控

node-exporter组件负责收集节点上的metrics监控数据，并将数据推送给prometheus, prometheus负责存储这些数据

grafana将这些数据通过网页以图形的形式展现

prometheus的架构图:

包含组件：

prometheus server: 主要负责数据采集和存储，提供promQL查询语言支持。prometheus是一个时序数据库，将采集到的监控数据按照时间序列的方式存储到本地磁盘。

Push Gateway: 支持临时性job主动推送指标的中间网关。

PromDash：使用rails开发的dashboard，用于可视化指标数据。

Exporters: 负责监控机器运行状态，提供被监控组件信息的 HTTP 接口被叫做 exporter。

直接采集： exporter内置了prometheus支持，直接向prometheus暴露数据端点。

间接采集：原不支持prometheus。通过prometheus提供的clien library编写的目标监控采集程序。

Altermanager: 从 Prometheus server 端接收到 alerts 后，会进行去除重复数据，分组，并路由到对收的接受方式，发出报警。常见的接收方式有：电子邮件，企业微信，钉钉 , webhook

WebUI: Prometheus内置一个简单的Web控制台，可以查询指标，查看配置信息或者Service Discovery等，实际工作中，查看指标或者创建仪表盘通常使用Grafana，Prometheus作为Grafana的数据源；9090提供图形化界面功能。

promethues 的各个组件基本都是用 golang 编写,对编译和部署十分友好.并且没有特殊依赖.基本都是独立工作。

工作原理：

Prometheus server 定期从配置好的 jobs 或者 exporters 中拉 metrics，或者接收来自 Pushgateway 发过来的 metrics，或者从其他的 Prometheus server 中拉 metrics。

Prometheus server 在本地存储收集到的 metrics，并运行已定义好的 alert.rules，记录新的时间序列或者向 Alertmanager 推送警报。

Alertmanager 根据配置文件，对接收到的警报进行处理，发出告警。

在图形界面中，可视化采集数据。

开始部署工作：

环境准备：K8S集群已部署kube-dns或者coredns

master/node节点环境部署

所有node节点下载监控所需镜像

# docker pull prom/node-exporter

# docker pull prom/prometheus:v2.0.0

# docker pull grafana/grafana:4.2.0

用daemonset方式部署 node-exporter

master操作

node-exporter.yaml

# mkdir k8s-prometheus

# cd k8s-prometheus

# cat node-exporter.yaml

---

apiVersion: apps/v1

kind: DaemonSet

metadata:

namespace: kube-system

labels:

k8s-app: node-exporter

spec:

selector:

matchLabels:

k8s-app: node-exporter

template:

metadata:

labels:

k8s-app: node-exporter

spec:

containers:

- image: prom/node-exporter

ports:

- containerPort: 9100

protocol: TCP

---

apiVersion: v1

kind: Service

metadata:

labels:

k8s-app: node-exporter

namespace: kube-system

spec:

ports:

- name: http

port: 9100

nodePort: 31672

protocol: TCP

type: NodePort

selector:

k8s-app: node-exporter

# kubectl apply -f node-exporter.yaml

部署prometheus组件

master操作

rbac-setup.yaml

# cat rbac-setup.yaml

apiVersion: /v1

kind: ClusterRole

metadata:

rules:

- apiGroups: [""]

resources:

- nodes

- nodes/proxy

- services

- endpoints

- pods

verbs: ["get", "list", "watch"]

- apiGroups:

- extensions

resources:

- ingresses

verbs: ["get", "list", "watch"]

- nonResourceURLs: ["/metrics"]

verbs: ["get"]

---

apiVersion: v1

kind: ServiceAccount

metadata:

namespace: kube-system

---

apiVersion: /v1

kind: ClusterRoleBinding

metadata:

roleRef:

apiGroup:

kind: ClusterRole

subjects:

- kind: ServiceAccount

namespace: kube-system

# kubectl apply -f rbac-setup.yaml

configmap的形式管理prometheus组件的配置文件

configmap.yaml

# cat configmap.yaml

apiVersion: v1

kind: ConfigMap

metadata:

namespace: kube-system

data:

prometheus.yml: |

global:

scrape_interval: 15s

evaluation_interval: 15s

scrape_configs:

- job_name: 'kubernetes-apiservers'

kubernetes_sd_configs:

- role: endpoints

scheme: https

tls_config:

ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

relabel_configs:

- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]

action: keep

regex: default;kubernetes;https

- job_name: 'kubernetes-nodes'

kubernetes_sd_configs:

- role: node

scheme: https

tls_config:

ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

relabel_configs:

- action: labelmap

regex: __meta_kubernetes_node_label_(.+)

- target_label: __address__

replacement: kubernetes.default.svc:443

- source_labels: [__meta_kubernetes_node_name]

regex: (.+)

target_label: __metrics_path__

replacement: /api/v1/nodes/${1}/proxy/metrics

- job_name: 'kubernetes-cadvisor'

kubernetes_sd_configs:

- role: node

scheme: https

tls_config:

ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

relabel_configs:

- action: labelmap

regex: __meta_kubernetes_node_label_(.+)

- target_label: __address__

replacement: kubernetes.default.svc:443

- source_labels: [__meta_kubernetes_node_name]

regex: (.+)

target_label: __metrics_path__

replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

- job_name: 'kubernetes-service-endpoints'

kubernetes_sd_configs:

- role: endpoints

relabel_configs:

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]

action: keep

regex: true

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]

action: replace

target_label: __scheme__

regex: (https?)

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]

action: replace

target_label: __metrics_path__

regex: (.+)

- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]

action: replace

target_label: __address__

regex: ([^:]+)(?::\d+)?;(\d+)

replacement: $1:$2

- action: labelmap

regex: __meta_kubernetes_service_label_(.+)

- source_labels: [__meta_kubernetes_namespace]

action: replace

target_label: kubernetes_namespace

- source_labels: [__meta_kubernetes_service_name]

action: replace

target_label: kubernetes_name

- job_name: 'kubernetes-services'

kubernetes_sd_configs:

- role: service

metrics_path: /probe

params:

module: [http_2xx]

relabel_configs:

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]

action: keep

regex: true

- source_labels: [__address__]

target_label: __param_target

- target_label: __address__

replacement: :9115

- source_labels: [__param_target]

target_label: instance

- action: labelmap

regex: __meta_kubernetes_service_label_(.+)

- source_labels: [__meta_kubernetes_namespace]

target_label: kubernetes_namespace

- source_labels: [__meta_kubernetes_service_name]

target_label: kubernetes_name

- job_name: 'kubernetes-ingresses'

kubernetes_sd_configs:

- role: ingress

relabel_configs:

- source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]

action: keep

regex: true

- source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]

regex: (.+);(.+);(.+)

replacement: ${1}://${2}${3}

target_label: __param_target

- target_label: __address__

replacement: :9115

- source_labels: [__param_target]

target_label: instance

- action: labelmap

regex: __meta_kubernetes_ingress_label_(.+)

- source_labels: [__meta_kubernetes_namespace]

target_label: kubernetes_namespace

- source_labels: [__meta_kubernetes_ingress_name]

target_label: kubernetes_name

- job_name: 'kubernetes-pods'

kubernetes_sd_configs:

- role: pod

relabel_configs:

- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]

action: keep

regex: true

- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]

action: replace

target_label: __metrics_path__

regex: (.+)

- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]

action: replace

regex: ([^:]+)(?::\d+)?;(\d+)

replacement: $1:$2

target_label: __address__

- action: labelmap

regex: __meta_kubernetes_pod_label_(.+)

- source_labels: [__meta_kubernetes_namespace]

action: replace

target_label: kubernetes_namespace

- source_labels: [__meta_kubernetes_pod_name]

action: replace

target_label: kubernetes_pod_name

# kubectl apply -f configmap.yaml

Prometheus deployment

prometheus.deploy.yml

# cat prometheus.deploy.yml

---

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

namespace: kube-system

spec:

replicas: 1

selector:

matchLabels:

app: prometheus

template:

metadata:

labels:

app: prometheus

spec:

containers:

- image: prom/prometheus:v2.0.0

command:

- "/bin/prometheus"

args:

- "--config.file=/etc/prometheus/prometheus.yml"

- "--storage.tsdb.path=/prometheus"

- "--storage.tsdb.retention=24h"

ports:

- containerPort: 9090

protocol: TCP

volumeMounts:

- mountPath: "/prometheus"

- mountPath: "/etc/prometheus"

resources:

requests:

cpu: 100m

memory: 100Mi

limits:

cpu: 500m

memory: 2500Mi

serviceAccountName: prometheus

volumes:

- name: data

emptyDir: {}

- name: config-volume

configMap:

# kubectl apply -f prometheus.deploy.yml

Prometheus service

prometheus.svc.yml

# cat prometheus.svc.yml

---

kind: Service

apiVersion: v1

metadata:

labels:

app: prometheus

namespace: kube-system

spec:

type: NodePort

ports:

- port: 9090

targetPort: 9090

nodePort: 30003

selector:

app: prometheus

#kubectl apply -f prometheus.svc.yml

部署 grafana 组件

grafana deployment

grafana-deploy.yaml

# mkdir ../grafana

# cd ../grafana

# cat grafana-deploy.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

namespace: kube-system

labels:

app: grafana

component: core

spec:

replicas: 1

selector:

matchLabels:

app: grafana

template:

metadata:

labels:

app: grafana

component: core

spec:

containers:

- image: grafana/grafana:4.2.0

imagePullPolicy: IfNotPresent

# env:

resources:

# keep request = limit to keep this container in guaranteed class

limits:

cpu: 100m

memory: 100Mi

requests:

cpu: 100m

memory: 100Mi

env:

# The following env variables set up basic auth twith the default admin user and admin password.

- name: GF_AUTH_BASIC_ENABLED

value: "true"

- name: GF_AUTH_ANONYMOUS_ENABLED

value: "false"

# - name: GF_AUTH_ANONYMOUS_ORG_ROLE

# value: Admin

# does not really work, because of template variables in exported dashboards:

# - name: GF_DASHBOARDS_JSON_ENABLED

# value: "true"

readinessProbe:

httpGet:

path: /login

port: 3000

# initialDelaySeconds: 30

# timeoutSeconds: 1

volumeMounts:

- name: grafana-persistent-storage

mountPath: /var

volumes:

- name: grafana-persistent-storage

emptyDir: {}

# kubectl apply -f grafana-deploy.yaml

grafana service

grafana-svc.yaml

# cat grafana-svc.yaml

apiVersion: v1

kind: Service

metadata:

namespace: kube-system

labels:

app: grafana

component: core

spec:

type: NodePort

ports:

- port: 3000

selector:

app: grafana

component: core

# kubectl apply -f grafana-svc.yaml

grafana ingress

grafana-ing.yaml

# cat grafana-ing.yaml

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

namespace: kube-system

spec:

rules:

- host: k8s.grafana

http:

paths:

- path: /

backend:

serviceName: grafana

servicePort: 3000

# kubectl apply -f grafana-ing.yaml

WEB界面：

查看node-exporte

http://任一节点iP:31672/metrics

prometheus对应的nodeport端口为30003

通过访问节点任一IP:30003/targets

可以看到prometheus已经成功连接上了k8s的apiserver（状态全部为UP）

通过端口进行granfa访问

先通过kubectl get svc -n kube-system 查看grafana 3000端口对应分到的nodeport

浏览器访问节点任一IP:nodeport 如下我分到的是nodeport是32539

默认用户名密码均为admin

#kubectl get svc -n kube-system

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

grafana NodePort 10.96.240.149 <none> 3000:32539/TCP 4h25m

kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 2d22h

metrics-server ClusterIP 10.96.138.32 <none> 443/TCP 44h

node-exporter NodePort 10.96.144.176 <none> 9100:31672/TCP 4h56m

prometheus NodePort 10.96.235.128 <none> 9090:30003/TCP 4h48m

grafana集成es grafana 集群_Powered by 金山文档_04

https://grafana.com/grafana/dashboards/

可获取模板id来创建监控模板，推荐个315先。

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。

上一篇：grep ps 查询进程号查看进程 linux grep

下一篇：esp连接wifi 密码错误回调无线esp是什么问题

提问和评论都可以，用心的回复会被更多人看到评论

发布评论

相关文章

官方博客	全部文章	热门标签	班级博客
了解我们	网站地图	意见反馈

鸿蒙开发者社区	51CTO学堂
51CTO	软考资讯

grafana集成es grafana 集群

grafana集成es grafana 集群

51CTO博客