Hello,
I have to combine logstash's lumberjack input, which in my case is the main shipper for all types of logs from centralized location to logstash and then I need to filter it with grok. So far so good - but the issue is that my logs are multiline.
I'm using the logstash filter Multiline and not the codec multiline as I have a single input of logs, and not all of my logs are multiline.
So my questions are:
- Can I have multime lumberjack inputs on the same port, with just separating by log type. At the moment my lumberjack input looks like this:
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/secure.crt"
ssl_key => "/etc/pki/tls/secure.key"
}
2. How to combine the grok and the multiline filter? So far I managed to get the multilines but I can not parse the message properly (by fields) - to every message I get _grokparsefailure.
My filter looks like this:
filter
{
if [type] == "applog" {
multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => "true"
what => "previous"
}
grok {
match => {
"message" => [ "%{APPLOG}" ] # this is my custom log pattern which you can find below
}
}
date {
match => ["apptimestamp","YYYY-MM-dd HH:mm:ss,SSS","ISO8601"]
target => "@timestamp"
locale => "en"
}
}
}
And %{APPLOG} equals to
%{TIMESTAMP_ISO8601:syslogtimestamp}%{SPACE}%{IPORHOST:servername}%{SPACE}%{GREEDYDATA:appender}:%{SPACE}%{TIMESTAMP_ISO8601:apptimestamp}%{SPACE}[%{NUMBER:thread}]%{SPACE}%{GREEDYDATA:appcomponent}\t%{GREEDYDATA:merchant}\t%{LOGLEVEL:loglevel}%{SPACE}%{NUMBER:appmaintrxid}%{SPACE}%{IPORHOST:servername}\t%{GREEDYDATA:appmessage}