Lumberjack input + grok and multiline


(Nikola Kolev) #1

Hello,

I have to combine logstash's lumberjack input, which in my case is the main shipper for all types of logs from centralized location to logstash and then I need to filter it with grok. So far so good - but the issue is that my logs are multiline.

I'm using the logstash filter Multiline and not the codec multiline as I have a single input of logs, and not all of my logs are multiline.

So my questions are:

  1. Can I have multime lumberjack inputs on the same port, with just separating by log type. At the moment my lumberjack input looks like this:

input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/secure.crt"
ssl_key => "/etc/pki/tls/secure.key"
}
2. How to combine the grok and the multiline filter? So far I managed to get the multilines but I can not parse the message properly (by fields) - to every message I get _grokparsefailure.

My filter looks like this:
filter
{
if [type] == "applog" {
multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => "true"
what => "previous"
}

grok {
	match => {
	"message" => [ "%{APPLOG}" ] # this is my custom log pattern which you can find below
         }
    }
date {
	match => ["apptimestamp","YYYY-MM-dd HH:mm:ss,SSS","ISO8601"]
	target => "@timestamp"
	locale => "en"
            }
    }

}

And %{APPLOG} equals to

%{TIMESTAMP_ISO8601:syslogtimestamp}%{SPACE}%{IPORHOST:servername}%{SPACE}%{GREEDYDATA:appender}:%{SPACE}%{TIMESTAMP_ISO8601:apptimestamp}%{SPACE}[%{NUMBER:thread}]%{SPACE}%{GREEDYDATA:appcomponent}\t%{GREEDYDATA:merchant}\t%{LOGLEVEL:loglevel}%{SPACE}%{NUMBER:appmaintrxid}%{SPACE}%{IPORHOST:servername}\t%{GREEDYDATA:appmessage}


(system) #2