Hi, i have two differetent resource log and i want to filtered in one server with one logstash. I've created the grok filter as I want it using the grol debugger and it's working and producing the appropriate output. But when I implement using logstash it doesn't work. Can anyone help?
It's the Logstash
input {
beats {
port => 5044
}
}
filter{
mutate {
rename => ["message", "rawMessage" ]
}
mutate {
gsub => ["rawMessage", "[\r\n\t]", ""]
gsub => ["rawMessage", "\\r", ""]
gsub => ["rawMessage", "\\n", ""]
gsub => ["rawMessage", "\\t", ""]
}
if[data_source] == "global" {
grok {
match => { "rawMessage" => "%{TIMESTAMP_ISO8601:datetime}\s\[%{GREEDYDATA:thread}\]\s%{LOGLEVEL:log}\s%{GREEDYDATA:category1}\s\s\-\s%{GREEDYDATA:message}"}
}
date {
match => ["datetime", "yyyy-MM-dd HH:mm:ss,SSS"]
target => "@timestamp"
}
}
else if[data_source] == "XMLDriver" {
grok {
patterns_dir => ["C:\Elasticsearch\ELK\Logstash\date_pattern.txt"]
match => { "rawMessage" => "%{GREEDYDATA:category1}\s-\s%{GREEDYDATA:category2}\s-\s%{GREEDYDATA:category3}\s-\s%{F_DATETIME_XML:datetime}\s-\s%{GREEDYDATA:category4}\s-\s%{GREEDYDATA:category5}\s-\s%{GREEDYDATA:message}"}
}
date {
match => [ "datetime" , "EEE MMM d HH:mm:ss" ]
target => "@timestamp"
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
it's the filebeat
filebeat.inputs:
- type: log
enabled: true
paths:
- C:\Elasticsearch\Log\ABC\globalserver2.log
- C:\Elasticsearch\Log\ABC\globalserver3.log
multiline.pattern: ^(\d{4})-(\d{2})-(\d{2})\s*(\d{2}):(\d{2}):(\d{2}),(\d{3})
multiline.negate: true
multiline.match: after
fields:
data_source: global
- type: log
enabled: true
paths:
- C:\Elasticsearch\Log\ABC\XMLdriver.log
multiline.pattern: ^R10\.97389
multiline.negate: true
multiline.match: after
fields:
data_source: XMLDriver
It's the log example
ServerGlobal Log
XMLDriver Log
