Allow logstash to store multi-line messages


(Sumit Vedpathak) #1

In my application, i am using filebeat->logstash->elasticsearch->kibana. Here filebeat is listening to a folder location(/XYZ/*.log)
This log file contains logs where messages can be in the form of multi-line.
For example, below are 3 log messages.

2018-05-07 04:30:00,954 [1] INFO WMK_Production.WMK_Production - Starting Welcome Kit production.
2018-05-07 04:30:01,579 [1] ERROR WMK_Production.WMK_Production - Unexpected error processing Welcome Kit production, cause: Specified argument was out of the range of valid values.
Parameter name: MaxDegreeOfParallelism.
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: MaxDegreeOfParallelism
at System.Threading.Tasks.ParallelOptions.set_MaxDegreeOfParallelism(Int32 value)
at WMK_Production.WMK_Production.Main() in D:..\Builds\SBS.Test.Build.WMK.Production\WMK_Production\WMK_Production.vb:line 106
2018-05-07 04:30:01,611 [1] DEBUG WMK_Production.WMK_Production - Sending notification email to: abc@zys.com, from: XYZ@ABC.com, on SMTP host: xyz.com.

In above 3 logs, 1st and 3rd logs are represented in a single line in log file, but 2nd log's error message is divided in multiple lines.

in my logstash conf file i am using below code in filter.

filter {
grok{
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} [%{NUMBER:number}] %{LOGLEVEL:loglevel} %{GREEDYDATA:file} - (?(.|\r|\n)*)"}
}

For single line log info, this works perfectly fine. But for 2nd log, filebeat itself is creating different events for each line, so ideally filebeat should return 3 events, but it is creating 8 events. 6 events for error log which ideally should send only 1. More over i need all new line log message as msg field.

My questions

  1. Where i need to make changes in order to get only 3 events, logstash or filebeat?
  2. What changes need to do in order to achieve this.

Thanks,
Sumit


(Paris Mermigkas) #2

You need to configure Filebeat appropriately to handle multiline messages when reading a file.

The examples listed there should help you get started.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.