Filebeat multiline codec not working in my case

[aarishramesh]

I have defined multiline codec in filebeat.yml like below

 multiline.pattern: '^%{TIMESTAMP_ISO8601} '
 multiline.negate: true
 multiline.match: after

But it does not seem to be working as multiple lines of log get appended together like below

Single line of log

2017-05-07 22:29:43 [0] [pool-2-thread-1] INFO  c.app.task.ChannelActiveCheckTask - ---- 
Inside checkIfChannelActive execution ----

The corresponding log stored in elastic search after multi-line parsing

---- Inside checkIfChannelActive execution ---- 2017-05-09 08:16:13 [0] [pool-2-thread-1] INFO  
XYZZ - XYZ :: 

XYZ 2017-05-09 08:16:13 [0] [pool-2-thread-1] INFO XYZ - XYZYZZ

Since the above did not work, I also tried using the below multi-pattern but it does not work too
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'

Below is my logstash.conf

input {
beats {
port => 5044
}
}

filter {
mutate {
gsub => ["message", "\n", " "]
}
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} [%{NOTSPACE:uid}] [%
{NOTSPACE:thread}] %{LOGLEVEL:loglevel} %{DATA:class}-%{GREEDYDATA:message}" ]
overwrite => [ "message" ]
}
date {
match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
target => "@timestamp"
}
if "_grokparsefailure" in [tags] {
drop { }
}
}

output {
elasticsearch {
hosts => localhost
index => "%{type}-%{+YYYY.MM.dd}"
}
}

Can someonehelp me fix this ? This parsing works fine on applying multiline codec in logstash.conf

[steffens (Steffen Siering)]

filebeat multiline pattern is not grok pattern as in logstash, but pure regular expressions. If you want to filter by date use:

multiline.pattern: '^\d{4}-\d{2}-\d{2} '

Note, grok like patterns are somewhat readable, but often having a more complex regex definition then just using a very simple regex. When doing multiline, you don't want to look at Content, but more at the shape of messages.

Do you have a more complete sample log for testing?

Also see this google playground for testing: https://play.golang.org/p/ES47bG6XN4

By community you can also find the filebeat-multiline-tester.

[steffens (Steffen Siering)]

Note: for testing I would start by disabling the filters in logstash and see filebeat is correctly creating the multiline events first (e.g. write events to file/console).