Multi line pattern for different services to one logstash

Elastic Stack Beats
1

I have installed file beat agents for different service and pointed them to Logstash and applied my filter for it but noticed multiple logs are coming in a single log even though i was using multi line pattern to differentiate my logs based on Timestamp.

sample log:
2018-10-24 11:39:50.815 [http-nio-8080-exec-9] [XXXX] [supportops@XXXX.com] INFO c.s.m.c.BusinessClassCodeController - getBusinessClassCodes :Retrieving Business Class Code by CodeType :business-class-one and size :: 1

filebeat configuration

multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
multiline.negate: true
multiline.match: after

Grok filter
input { beats
{
port => 5044
}
}

filter {
grok {

match => [ "message","(?<timestamp>%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}[/ ]%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))\s+(\[%{USERNAME:thread}\])\s+(\[%{USERNAME:tenentId}\])\s+(\[(?<userId>%{WORD}+@[0-9A-Za-z][0-9A-Za-z-]{0,62}+\w+.[a-z]{0,62})\]) %{WORD:loglevel}  %{JAVACLASS:classname} %{GREEDYDATA:logmessage}"]

match => [ "message","(?<timestamp>%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))\s+(\[%{USERNAME:thread}\])\s+(\[%{SPACE}])\s+(\[%{SPACE}])%{SPACE}%{WORD:loglevel}%{SPACE}%{JAVACLASS:classname}%{SPACE}[/-]%{GREEDYDATA:logmessage}"]

}

mutate { remove_field => [ "[beat][name]", "[host][name]", "[beat][version]", "offset","[input][type]"] }

}

output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }

}

Filter is applied successfully but i am seeing multiple logs i.e next logs till an extent are attached as greedy data as below.

Output from kibana only logmessage is given hear

logmessage : - getDayparts :Retrieving all DayPart :: 13
2018-10-24 11:39:50.815 [http-nio-8080-exec-9] [XXXX] [supportops@XXXX.com] INFO c.s.m.c.BusinessClassCodeController - getBusinessClassCodes :Retrieving Business Class Code by CodeType :business-class-one and size :: 1
2018-10-24 11:39:51.010 [http-nio-8080-exec-1] [XXXX] [supportops@XXXX.com] INFO c.s.m.c.BusinessClassCodeController - getBusinessClassCodes :Retrieving Business Class Code by CodeType :business-class-three and size :: 1

2

@kuna If I understand correctly the multiline is not correctly applied to the events and you get two different event in a single event?

3

@pierhugues might be correct!! BTW for java stack trace it is working fine.. their i don't see any issue..

4

One thing that strikes me is the pattern that you are defining is not anchored regular expression.

 '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'

Adding ^ at the beginning force it to look up the pattern at the beginninge of the string.

5

@pierhugues i used the same pattern but still its not working ..

6

@kuna can you create a minimal test case + config? In this case that would be a fake log file with multiple entries that get incorrectly interleaved by filebeat.

That would be helpful in debugging this.

7

@kuna You could also use https://github.com/hartfordfive/filebeat-multiline-tester or this playground https://play.golang.org/p/uAd5XHxscu to test your patterns.

8

@Andrew_Cholakian1 when i tried running one application and used beats to push the logs their i dint see any issue but when i run multiple services using multiple beats its not working in the same way..

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.