采集ingress日志 filebeat采集日志直接到es

转载

网络安全侠 2024-03-28 07:54:41

文章标签 采集ingress日志 elk nginx tomcat json 文章分类 云原生云计算

背景

分析--->报警	Kibana
	展示			Kibana
	存储			ElasticSerach
	过滤			Logstash
	收集			Filebeat, Logstash

logstash：因为是由java写的，所以在它收集日志时非常占用业务系统的资源，从而影响线上业务，所以我们将其替换为filebeat.
filebeat：为轻量的日志收集组件，会让业务系统的运行更加的稳定.


由于痛点：
	1.出现故障,要排查的日志非常的多,没有办法很快定位;
	2.统计TOP,PV,UV,IP信息; awk无法满足需求;  | elkstack | [ 在线活跃用户, 订单交易金额; ]
所以出现了日志收集: logstash 或者 filebeat

收集日志到es(elasticsearch)存储上有两种方法：
	1.直接用filebeat将日志格式转为json格式
	2.上面方法不可用，用filebeat收集日志，接入logstash进行处理，再收集到es存储


PS：
	app日志中，若是出现了如下类型的，我们的filebeat不能够直接修改日志格式将其修改为json格式，下面会解决.
	例如：
		[商品名] [用户] [订单数] [交易金额]

Nginx篇

虚拟主机收集+Nginx错误日志收集

#nginx主配置文件
[root@web01 ~]# cat /etc/nginx/nginx.conf 
user  www;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  10000;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

log_format json '{ "time_local": "$time_local", '
						  '"remote_addr": "$remote_addr", '
						  '"referer": "$http_referer", '
						  '"request": "$request", '
						  '"status": $status, '
						  '"bytes": $body_bytes_sent, '
						  '"agent": "$http_user_agent", '
						  '"x_forwarded": "$http_x_forwarded_for", '
						  '"up_addr": "$upstream_addr",'
						  '"up_host": "$upstream_http_host",'
						  '"upstream_time": "$upstream_response_time",'
						  '"request_time": "$request_time"'
	'}';

    access_log  /var/log/nginx/access.log  json;
    keepalive_timeout  65;
    include /etc/nginx/conf.d/*.conf;
}


#网站配置文件
[root@web01 ~]# cat /etc/nginx/conf.d/elk.qian.com.conf 
server {
	listen 80;
	server_name elk.qian.com;
	root /data;
	access_log /var/log/nginx/elk.log json;

	location / {
		index index.html;
	}

}


[root@web01 ~]# cat /etc/nginx/conf.d/blog.ming.com.conf 
server {
	listen 80;
	server_name blog.ming.com;
	root /code;
	access_log /var/log/nginx/blog.log json;

	location / {
		index index.html;
	}

}



#filebeat配置文件
[root@web01 ~]# cat /etc/filebeat/nginx.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/elk.log
  json.keys_under_root: true  #Flase会将json解析的格式存储至messages，改为true则不存储至message
  json.overwrite_keys: true   #覆盖默认message字段，使用自定义json格式的key
  tags: nginx_elk

- type: log
  enabled: true
  paths:
    - /var/log/nginx/blog.log
  json.keys_under_root: true  #Flase会将json解析的格式存储至messages，改为true则不存储至message
  json.overwrite_keys: true   #覆盖默认message字段，使用自定义json格式的key
  tags: nginx_blog

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  json.keys_under_root: true  #Flase会将json解析的格式存储至messages，改为true则不存储至message
  json.overwrite_keys: true   #覆盖默认message字段，使用自定义json格式的key
  tags: nginx_error


output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
  indices:
    - index: "nginx-elk-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "nginx_elk"
    - index: "nginx-blog-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "nginx_blog"
    - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "nginx_error"



setup.ilm.enabled: false
setup.template.name: "nginx"       #定义模板名称
setup.template.pattern: "nginx-*"  #定义模板的匹配索引名称

#setup.template.settings:		#开启自定义分片数量，不建议开启
#  index.number_of_shards: 3	#主分片数为：3
#  index.number_of_replicas: 1	#副本数为：1

Tomcat篇+错误日志收集

#Tomcat主配置文件
[root@web01 ~]# cat /soft/tomcat/conf/server.xml 
........此处省略配置信息
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common" -->
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t "%r" %s %b" />

      </Host>

      <Host name="tomcat.blog.com"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
                <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="blog_tomcat_access_log" suffix=".txt"
               pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}" />

      </Host>

    </Engine>
  </Service>
</Server>



Filebeat配置文件
[root@web01 ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /soft/tomcat/logs/blog_tomcat_access_log.*.txt
  json.keys_under_root: true  #Flase会将json解析的格式存储至messages，改为true则不存储至message
  json.overwrite_keys: true   #覆盖默认message字段，使用自定义json格式的key
  tags: tomcat_blog

- type: log
  enabled: true
  paths:
    - /soft/tomcat/logs/catalina.out
  tags: tomcat_error
  multiline.pattern: '^\d{2}'		#此处应注意，因为Tomcat的错误日志不同于网站日志，不能将其改为json格式，可用此配置，具体可以 https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html 查看官网
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 1000


output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]
  indices:
    - index: "tomcat_blog-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "tomcat_blog"
    - index: "tomcat_error-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.contains:
        tags: "tomcat_error"




setup.ilm.enabled: false
setup.template.name: "tomcat"       #定义模板名称
setup.template.pattern: "tomcat-*"  #定义模板的匹配索引名称

系统篇

#用rsyslog工具将系统配置集中到一个日志中去，由filebeat收集

[root@web01 ~]# yum install rsyslog -y
[root@web01 ~]# vim /etc/rsyslog.conf		#修改rsyslog配置文件
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
*.*     /var/log/ming.log			#添加
[root@web01 ~]# systemctl start rsyslog




#filebeat配置文件
#基本配置
[root@web01 ~]# cat  /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/text.log

output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]


#增强版配置
[root@web01 ~]# cat  /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/text.log

  include_lines: ['^ERR', '^WARN', 'sshd']		  # 仅包含,错误信息,警告信息,sshd的相关配置,其他的都会过滤掉

output.elasticsearch:
  hosts: ["10.0.0.161:9200","10.0.0.162:9200","10.0.0.163:9200"]

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。