elasticsearch+filebet+kabana收集日志

一.Filebeat介绍

Filebeat附带预构建的模块,这些模块包含收集、解析、充实和可视化各种日志文件格式数据所需的配置,每个Filebeat模块由一个或多个文件集组成,这些文件集包含摄取节点管道、Elasticsearch模板、Filebeat勘探者配置和Kibana仪表盘。

Filebeat模块很好的入门,它是轻量级单用途的日志收集工具,用于在没有安装java的服务器上专门收集日志,可以将日志转发到logstash、elasticsearch或redis等场景中进行下一步处理

为什么使用filebeat?

filebeat比logstash占用更少的系统资源,特别是内存。

二.使用filebeat收集nginx日志

2.1使用filebeat收集普通的nginx日志

1.安装Nginx

cat >/etc/yum.repos.d/nginx.repo <<EOF
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF

yum install nginx -y 
systemctl start nginx 
curl 127.0.0.1

2.配置Nginx并创建测试页面

rm -rf /etc/nginx/conf.d/default.conf 
cat >/etc/nginx/conf.d/www.conf<<EOF
server {
    listen       80;
    server_name  localhost;
    location / {
        root   /code/www;
        index  index.html index.htm;
    }
}
EOF
mkdir /code/www/ -p
echo "db01-www" > /code/www/index.html
nginx -t
systemctl restart nginx
curl 127.0.0.1
tail -f /var/log/nginx/access.log

3.安装filebet

rpm -ivh filebeat-6.6.0-x86_64.rpm

4.配置filebeat

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
EOF

5.启动测试

systemctl start filebeat

6.检查结果

tail -f /var/log/filebeat/filebeat
curl -s 127.0.0.1:9200/_cat/indices|awk '{print $3}'

7.es-head查看

8.kabana查看


说明:这样收集的日志,信息全是在messge这个字段,还是无法分离我们想要查看的内容。

2.2.filebeat收集Nginx的json格式日志

1.修改nginx配置文件使日志转换成json

log_format json '{ "time_local": "$time_local", '
                          '"remote_addr": "$remote_addr", '
                          '"referer": "$http_referer", '
                          '"request": "$request", '
                          '"status": $status, '
                          '"bytes": $body_bytes_sent, '
                          '"agent": "$http_user_agent", '
                          '"x_forwarded": "$http_x_forwarded_for", '
                          '"up_addr": "$upstream_addr",'
                          '"up_host": "$upstream_http_host",'
                          '"upstream_time": "$upstream_response_time",'
                          '"request_time": "$request_time"'
    ' }';
    access_log  /var/log/nginx/access.log  json;

2.清除旧日志

> /var/log/nginx/access.log

3.检查并重启nginx

nginx -t 
systemctl restart nginx

4.修改filebeat配置文件支持json解析
说明:由于filebeat是go语言开发的,所以默认的是不支持json解析的,需要额外配置。配置如下:

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
EOF

5.删除ES里以前的索引

es-head >> filebeat-6.6.0-2019.11.15 >> 动作 >>删除 

6.重启filebeat

systemctl restart filebeat

7.es-head查看


8.kabana查看
没有配置filebeat的结果为:


配置的结果为:


2.3.filebeat自定义ES索引名称

1.理想的索引的名称要与收集的对象的日志相关,以便区分。
例如 nginx-6.6.0-2020.02
2.filebeat配置

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF 

3.测试访问后es-head查看


4.kabana添加新的模板查看

2.4.filebeat按照服务类型拆分索引

1.两种配置方法

1.第一种写法

cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        source: "/var/log/nginx/access.log"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        source: "/var/log/nginx/error.log"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF 

2.第二种写法:

cat >/etc/filebeat/filebeat.yml<<EOF 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"
    - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

2.重启filebeat

systemctl restart filebeat

3.es-head查看


4.kabana查看结果


2.5.多服务器收集相同的日志合并

说明:默认是会自动合并
新开一台服务器,配置与之前的一样。
1.es-head查看结果


2.kabana查看结果
一起查询


分离查询所需查看的结果

2.6.使用filebeat模块收集nginx日志

官方配置说说明请参考:https://www.elastic.co/guide/en/beats/filebeat/6.6/filebeat-module-nginx.html

说明:之前的收集需要修改nginx日志格式为json以及配置filebeat支持解析json格式,这样的使用适合新的环境进行使用,如果后期才上线日志收集这一块,使用filebeat模块收集日志可以不用修改原有的普通日志的格式和修改filebeat的配置文件。

0.配置es支持nginx模块的插件

cd /usr/share/elasticsearch/
./bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip 
./bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip
systemctl restart elasticsearch

1.配置filebeat配置文件,配置支持模块功能

#默认配置是faslse
============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s
#精简并修改配置文件为ture

[root@nginx ~]# vim /etc/filebeat/filebeat.yml
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
  - index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      event.dataset: "nginx.access"
  - index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      event.dataset: "nginx.error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true

2.查看支持的模块

[root@nginx ~]# filebeat modules list
Enabled:   #表示已经开启的模块

Disabled:  #表示所有支持的模块或未开启的模块
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
suricata
system
traefik

3.开启nginx模块

[root@nginx ~]# filebeat modules enable nginx
Enabled nginx
[root@nginx ~]# filebeat modules list
Enabled:
nginx

Disabled:
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
osquery
postgresql
redis
suricata
system
traefik

4.配置nginx模块

#默认格式
[root@nginx /etc/filebeat]# vim modules.d/nginx.yml
- module: nginx
  # Access logs
  access:
     denabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
#修改并简化配置文件
[root@nginx /etc/filebeat/modules.d]# vim nginx.yml
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log"]

  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log"]

5.启动filebeat

systemctl start filebeat.service

6.es-head查看结果


7.kabana界面创建索引模板并查看



说明:模块中字段拆分的更加详细

三.使用filebeat收集tomccat日志

1.修改tomact的日志文件问json

#默认格式
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b" />
#修改为json格式
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt" pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quo
t;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quo
t;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;Sen
dBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%
{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/>

2.filebeat配置文件设置

[root@tomcat ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/tomcat/logs/localhost_access_log.*.txt
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["tomcat"]

output.elasticsearch:
  hosts: ["10.0.0.72:9200"]
  index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM}"

setup.template.name: "tomcat"
setup.template.pattern: "tomcat_*"
setup.template.enabled: false
setup.template.overwrite: true

3.重启filebeat

[root@tomcat ~]# systemctl restart filebeat

4.测试访问查看es-head



5.kabana


四. filebeat收集java多行匹配模式

官方配置请参考链接:https://www.elastic.co/guide/en/beats/filebeat/6.6/multiline-examples.html

1.filebeat配置文件

cat >/etc/filebeat/filebeat.yml<<EOF   
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/elasticsearch/elasticsearch.log 
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  index: "es-%{[beat.version]}-%{+yyyy.MM}"
    
setup.template.name: "es"
setup.template.pattern: "es-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

2.重启filebeat

systemctl restart filebeat

五.使用filebeat模块收集mysql慢日志和错误日志

1.配置mysql错误日志和慢日志路径
编辑my.cnf

vim /etc/my.cnf
[mysqld]
slow_query_log=ON
slow_query_log_file=/data/mysql/data/slow.log
long_query_time=1

2.重启mysql并制造慢日志

systemctl restart mysql 

慢日志制造语句

select sleep(2) user,host from mysql.user ;

3.确认慢日志和错误日志确实有生成

mysql -uroot -poldboy123 -e "show variables like '%slow_query_log%'"

4.激活filebeat的mysql模块

filebeat module enable mysql

5.配置mysql的模块

[root@db05 ~]# vim /etc/filebeat/modules.d/mysql.yml 
- module: mysql
  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/application/mysql/data/error.log"]

  # Slow logs
  slowlog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/application/mysql/data/slow.log"]

6.配置filebeat根据日志类型做判断

cat >/etc/filebeat/filebeat.yml<<EOF 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true 
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
  - index: "mysql-slow-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      source: "/application/mysql/data/slow.log"
  - index: "mysql-err-%{[beat.version]}-%{+yyyy.MM}"
    when.contains:
      source: "/application/mysql/data/error.log"

setup.template.name: "mysql"
setup.template.pattern: "mysql-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

7.重启filebeat

systemctl restart filebeat

8.es-head查看

六.使用filebeat收集docker日志

filebeat收集docker日志终极杀人王火云邪神版

1.需求分析
json格式并且按照下列索引生成
docker-nginx-access-6.6.0-2020.02
docker-db-access-6.6.0-2020.02
docker-db-error-6.6.0-2020.02
docker-nginx-error-6.6.0-2020.02

3.创建新容器并挂载本地的目录到容器的日志文件目录下

docker run -d -p 80:80 -v /opt/nginx:/var/log/nginx nginx
docker run -d -p 8080:80 -v /opt/mysql:/var/log/nginx nginx

4.准备json格式的nginx配置文件并拷贝到容器里并重启

docker cp nginx.conf 5d62b35651e6:/etc/nginx/
docker cp nginx.conf 310e85addbcd:/etc/nginx/
docker stop $(docker ps -qa)
docker start Nginx容器的ID
docker start mysql容器的ID

5.配置filebeat配置文件

cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log 
  enabled: true
  paths:
    - /opt/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["nginx_access"]

- type: log 
  enabled: true
  paths:
    - /opt/nginx/error.log
  tags: ["nginx_err"]

- type: log 
  enabled: true
  paths:
    - /opt/mysql/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["db_access"]

- type: log 
  enabled: true
  paths:
    - /opt/mysql/error.log
  tags: ["db_err"]

output.elasticsearch:
  hosts: ["10.0.0.51:9200"]
  indices:
    - index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "nginx_access"

    - index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "nginx_err"

    - index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "db_access"

    - index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "db_err"

setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF

6.启动filebeat

systemctl restart filebeat

7.访问并测试

curl 127.0.0.1
curl 127.0.0.1:8080/
cat /opt/nginx/access.log
cat /opt/mysql/access.log

8.es-head查看


禁止转载,如需转载请通过简信或评论联系作者。

推荐阅读更多精彩内容