已经验证的几个方法：
https://www.elastic.co/guide/en/logstash/current/first-event.html
1. filebeat
client:
[elastic@cms104 filebeat-6.0.0-linux-x86_64]$ vi filebeat.yml filebeat.prospectors: - type: log paths: - /opt/glassfish/domains/domain1/logs/server.log output.logstash: hosts: ["10.2.28.8:5044"]
server:
[elastic@localhost bin]$ vi first-pipeline.conf
input {
beats { port => "5044" } } # The filter part of this file is commented out to indicate that it is # optional.
# filter { # # }
output { stdout { codec => rubydebug } }
可以成功收到log:
2. elasticsearch config
server
input { file { id => "my_file_plugin" #Default value is "plain" codec => json #type => "log" path => "/home/elastic/kibana-6.0.0-linux-x86_64/nohu*.out" #/var/log/**/*.log, a recursive search of /var/log will be done for all *.log files discover_interval => 10 #start_position => "beginning" #defalut end } }
filter { if ([type] != "response") { drop {} } if ([req][remoteAddress] == "10.2.1.66") { drop {} } mutate { add_field => { "real_ip" => "79.1.14.87" } add_field => { "new_ip" => "%{[req][remoteAddress]}" } remove_field => ["[req][referer]"] } geoip { source => "real_ip" } # grok { match => ["message", "%{USERNAME:username}"]} #date { # match => [ "logdate", "MMM dd yyyy HH:mm:ss" ] #} }
output { stdout { codec => rubydebug } elasticsearch { # Indexes may not contain uppercase characters index => "log-kibana-%{+YYYY.MM.dd}" hosts => ["10.2.28.8:9200"] #, "10.2.135.104:9200"] user => "elastic" password => "password" } }
Grok是个比较强大的小工具
https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
字符必须一一对应
------------------------------------------------------
123,456 firstStr secStr thridStr fourthStr
%{NUMBER:duration}\,%{NUMBER:speed} %{DATA:method} %{GREEDYDATA:logContent} %{WORD:lastFld}
{
"duration": "123",
"method": "firstStr",
"logContent": "secStr thridStr",
"lastFld": "fourthStr",
"speed": "456"
}
官方sample
log:
192.168.10.97 - - [19/Jul/2016:16:28:52 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"
Grok:
%{IPORHOST:addre} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:http_method} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:status} (?:%{NUMBER:bytes}|-) \"(?:%{URI:http_referer}|-)\" \"%{GREEDYDATA:User_Agent}\"
result:
{ "request": "/", "auth": "-", "ident": "-", "User_Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36", "addre": "192.168.10.97", "http_method": "GET", "bytes": "23", "httpversion": "1.1", "timestamp": "19/Jul/2016:16:28:52 +0800", "status": "200" }
