Elasticsearch相关软件安装
1.1 Elasticsearch相关软件下载
我这里elasticsearch选择二进制安装，其余使用rpm进行安装，以下连接可进行相应版本下载
kibana-7-6-1、elasticsearch-7-6-1、logstash-7-6-1、filebeat-7-6-1、node-v14.17.6-linux-x64.tar.xz
https://www.elastic.co/cn/downloads/past-releases/kibana-7-6-1 下载相应版本
https://www.elastic.co/cn/downloads/past-releases/elasticsearch-7-6-1 下载相应版本
https://www.elastic.co/cn/downloads/past-releases/logstash-7-6-1 下载相应版本
https://www.elastic.co/cn/downloads/past-releases/filebeat-7-6-1 下载相应版本
https://nodejs.org/zh-cn/download/ 下载相应版本
注意：elasticsearch-7版本以后不许要单独安装JDK环境，elasticsearch二进制安装启动需要普通用户，root用户启动会报错
1.2 Elasticsearch单机部署
1.启动需要使用普通用户并创建相关文件目录
groupadd es
useradd es -g es
mkdir -p /usr/local/elasticsearch/
tar xf elasticsearch-7.6.1-linux-x86_64.tar.gz -C /usr/local/elasticsearch/
2.修改文件权柄
vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535
3.修改java环境变量
3.修改java环境变量
vim /usr/local/elasticsearch/elasticsearch-7.6.1/bin/elasticsearch-env
JAVA_HOME={安装目录下的}elasticsearch/jdk #elasticsearch-env脚本文件中添加JAVA环境变量，本环境变量目录为elasticsearch下的JDK
4.vim /etc/sysctl.conf
vm.max_map_count = 262144
sysctl -p #加载下
5.修改配置文件
vim /usr/local/elasticsearch/elasticsearch-7.6.1/conf/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0 ---- 开启
discovery.seed_hosts: ["10.0.0.200"] ---开启
cluster.initial_master_nodes: ["10.0.0.200"] ----开启
6.jav.options配置文件
vim /usr/local/elasticsearch/elasticsearch-7.6.1/conf/jav.options
由于UseConcMarkSweepGC 以后过时了,需要修改
8-13:-XX:+UseConcMarkSweepGC 修改为 8-13:-XX:+UseG1GC
7.授权目录
chown -R es. 安装目录
chown -R es. /usr/local/elasticsearch/
8.启动并测试
[root@ybj /usr/local/elasticsearch]# elasticsearch-7.6.1/bin/elasticsearch -d
[root@ybj /usr/local/elasticsearch]# curl 10.0.0.200:9200
1.3 Elasticsearch集群部署
需要修改配置文件，其他的和单机一样
vim /usr/local/elasticsearch/elasticsearch-7.6.1/conf/elasticsearch.yml
cluster.name: elk # 集群的农资
node.name: node-1 # 节点名称和cluster.initial_master_nodes 这里的名字需要对应
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.0.0.200", "10.0.0.201", "10.0.0.202"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
discovery.zen.ping_timeout: 60s
#跨域请求配置(为了让类似head的第三方插件可以请求es)
http.cors.enabled: true
http.cors.allow-origin: "*"
查看集群状态
http://10.0.0.200:9200/_cluster/health?pretty
1.4 安装可视化工具elasticsearch head插件
1.下载node https://nodejs.org/zh-cn/download/ 官网网站,可以选择版本
wget https://nodejs.org/dist/v14.17.6/node-v14.17.6-linux-x64.tar.xz
2.解压并 添加环境变量
tar xf node-v14.17.6-linux-x64.tar.xz -C /usr/local/node/
vim /etc/profile
export NODE_HOME=/usr/local/node/node-v14.17.6-linux-x64
export PATH=$PATH:$NODE_HOME/bin
source /etc/profile
3.测试命令并安装
node -v
npm -v
npm install -g cnpm --registry=https://registry.npm.taobao.org
4.创建插件目录
mkdir -P /usr/local/elasticsearch/plugins
cd /usr/local/elasticsearch/plugins
5.拉取elasticsearch-head代码
git clone git://github.com/mobz/elasticsearch-head.git
6..进入目录修改配置文件,添加参数
vim Gruntfile.js
搜索:server
server: {
options: {
hostname:'*', 添加此处即可,不让无法访问
port: 9100,
base: '.',
keepalive: true
7 安装依赖
cd /usr/local/elasticsearch/plugins/elasticsearch-head
cnpm install
8.启动
cnpm run start &cnpm run start &
1.5 安装kibana
1.下载kibana ，可以选择rmp进行安装
https://www.elastic.co/cn/downloads/past-releases/kibana-7-6-1 下载相应版本
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.6.1-linux-x86_64.tar.gz
2.创建kibana目录并解压
mkidr -p /usr/local/kibana
tar xf kibana-7.6.1-linux-x86_64.tar.gz -C /usr/local/kibana
3.修改配置
vim kibana.yaml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://10.0.0.200:9200","http://10.0.0.201:9201","http://10.0.0.202:9202"]
4启动
cd /usr/local/kibana/kibana-7.6.1-linux-x86_64/bin #进入bin目录
./kibana --allow-root
1.6filebeat安装 可以使用rpm包的方式
下载 https://www.elastic.co/cn/downloads/past-releases/filebeat-7-6-1
rpm -ivh https://www.elastic.co/cn/downloads/past-releases/filebeat-7-6-1
注意：集群搭建好，很多时候都要去修改filebeat 配置文件，注意缩进
自定义索引
filebeat.yml 修改自定义索引
index: "nginx-%{[agent.version]}-%{+yyyy.MM}"
setup.ilm.enabled: false
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: tru
1.7filebeat收集Nginx的json格式日志
上面方案不完善的地方
所有日志都存储在message的value里,不能拆分单独显示
理想中的情况
可以把日志所有字段拆分出来
{
$remote_addr : 192.168.12.254
- : -
$remote_user : -
[$time_local]: [10/Sep/2019:10:52:08 +0800]
$request: GET /jhdgsjfgjhshj HTTP/1.0
$status : 404
$body_bytes_sent : 153
$http_referer : -
$http_user_agent :ApacheBench/2.3
$http_x_forwarded_for:-
}
修改nginx配置文件使日志转换成json
log_format json '{ "time_local": "$time_local", '
'"remote_addr": "$remote_addr", '
'"referer": "$http_referer", '
'"request": "$request", '
'"status": $status, '
'"bytes": $body_bytes_sent, '
'"agent": "$http_user_agent", '
'"x_forwarded": "$http_x_forwarded_for", '
'"up_addr": "$upstream_addr",'
'"up_host": "$upstream_http_host",'
'"upstream_time": "$upstream_response_time",'
'"request_time": "$request_time"'
清除旧日志
> /var/log/nginx/access.log
检查并重启nginx
nginx -t
systemctl restart nginx
1.8nginx转换成json之后仍然不完善的地方
通过查看发现,虽然nginx日志变成了json,但是es里还是存储在message里仍然不能拆
7.修改filebeat配置文件支持json解析
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true #增加
json.overwrite_keys: true #增加
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
EOF
1.9 filebeat配置
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
2.0 filebeat按照服务类型拆分索引
1.第一种写法
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/var/log/nginx/access.log"
- index: "nginx-error-%{[ban.version]}-%{+yyyy.MM}"
when.contains:
source: "/var/log/nginx/error.log"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
2.第二种写法：
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "access"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
Filebeat收集tomcat的json日志
[root@web01 ~]# /opt/tomcat/bin/shutdown.sh
[root@web01 ~]# sed -n '162p' /opt/tomcat/conf/server.xml
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
2 启动tomcat
/opt/tomcat/bin/startup.sh
3.配置filebeat
cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/tomcat/logs/localhost_access_log.*.txt
json.keys_under_root: true
json.overwrite_keys: true
tags: ["tomcat"]
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "tomcat"
setup.template.pattern: "tomcat_*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
4.重启filebeat
systemctl restart filebeat
5.filebeat收集java多行匹配模式
2.1filbeat使用模块收集nginx日志
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
22.激活filebeat模块并查看
filebeat modules list 查看
filebeat modules enable nginx 激活 实际上就
output.elasticsearch:
hosts: ["10.0.0.200"]
index: "nginx-%{[agent.version]}-%{+yyyy.MM}"
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "nginx"
setup.template.pattern: "nginx-*
24.进入kinaba页面进行测试
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
