Issue with Multiline Logs

[Vivek_Sharma]

Hi

I am trying to read log file with multilines for a single log.

I have configured filebeat.yml as below:

> - input_type: log 
>   paths:
>     - C:\ELK\*.log
>   fields: 
>     tech_stack: XXX
>     kpi_type: YYY
> 
>   multiline.pattern: '^([Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec])\w+\s(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\s(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)'
>   multiline.negate: true
> 
>   multiline.match: after

Logstash has been defined as below:

> input {
>   beats {
>     port => 5044
>   }
>    
> }
> 
>   filter {
>   grok {
>     match => { "message" => [
> #	For reading STO log files. This will be common for both start and stop service success scenario
> 	"%{SYSLOGTIMESTAMP:timestamp} \[Host:%{HOSTNAME:host_name}\:-1\|Service:%{PROG:log_service_name}\|Context:%{USERNAME:context}\|Session:%{USERNAME:session_id}\|User:%{USERNAME:user_name}\|ApplicationId:%{USERNAME:application_id}\|MessageId:%{USERNAME:message_id}\|CorrelationId:%{USERNAME:correlation_id}\|GeneratingSystem:%{USERNAME:generating_system}\|LogMessage:%{GREEDYDATA:log_message}\|MessageBody:STO_NUMBER:%{USERNAME:sto_number}]\|"
> 	] }
>   }
> 	
>     mutate {
>       add_field => { "tech_stack" => "%{[fields][tech_stack]}" }
> 	  add_field => { "kpi_type" => "%{[fields][kpi_type]}" }
>     }
> }
> 
> output {
>   elasticsearch {
>     hosts => "localhost:9200"
> 	user => "elastic"
> 	password => "changeme"
>     manage_template => false
> 	index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
>     document_type => "%{[@metadata][type]}" 
>   }
>    stdout { codec => rubydebug }
> }

Now my logs are as below:

Jul 29 19:16:43 [Host:dlap-w1is0419.xxxxxxxxx.com:-1|Service:RPU_WWW_RT_CC_Utils.LoggingFramework:logMessage|
Context:a9604140-f805-1469-9704-fffffffaa84d|Session:null|User:Administrator|ApplicationId:RT_IA1424|MessageId:null|CorrelationId:d4e01dbb-17f8-4851-b54b-31074a4a57fb|GeneratingSystem:webMethods|
LogMessage:Service Start :RPU_WWW_RT_PRP_OrderDetails.Maps:mapXXXXXXXXXXXSvc|MessageBody:XXX_NUMBER:1245789654]|

As per filebeat configuration, these three lines of log need to be merged into a single line and then passed to logstash. But logstash is receiving all these logs in separate lines.

Could anyone please help.

[steffens (Steffen Siering)]

Multiline support is based on regular expression. As the [<chars>] term creates a custom character class, I don't think [Jan|Feb|...] is doing what you'd expect. Use (...) to create an 'or' group. Like (Jan|Feb|...). You can try to shorten the regex to this (the expression should capture a structural pattern, no need to be too strict on actual content): '^[JFMASOND][a-z]{2} \d{2}:\d{2}:\d{2} \['

Btw. since filebeat 5.3 (I think) you can use () to build groups. The matcher in libbeat applies some simple optimizations to the pattern. Like internally optimizing (<terms>) to (?:<terms>).