In my application, i am using filebeat->logstash->elasticsearch->kibana. Here filebeat is listening to a folder location(/XYZ/*.log)
This log file contains logs where messages can be in the form of multi-line.
For example, below are 3 log messages.
2018-05-07 04:30:00,954 [1] INFO WMK_Production.WMK_Production - Starting Welcome Kit production.
2018-05-07 04:30:01,579 [1] ERROR WMK_Production.WMK_Production - Unexpected error processing Welcome Kit production, cause: Specified argument was out of the range of valid values.
Parameter name: MaxDegreeOfParallelism.
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: MaxDegreeOfParallelism
at System.Threading.Tasks.ParallelOptions.set_MaxDegreeOfParallelism(Int32 value)
at WMK_Production.WMK_Production.Main() in D:..\Builds\SBS.Test.Build.WMK.Production\WMK_Production\WMK_Production.vb:line 106
2018-05-07 04:30:01,611 [1] DEBUG WMK_Production.WMK_Production - Sending notification email to: abc@zys.com, from: XYZ@ABC.com, on SMTP host: xyz.com.
In above 3 logs, 1st and 3rd logs are represented in a single line in log file, but 2nd log's error message is divided in multiple lines.
in my logstash conf file i am using below code in filter.
filter {
grok{
match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} [%{NUMBER:number}] %{LOGLEVEL:loglevel} %{GREEDYDATA:file} - (?(.|\r|\n)*)"}
}
For single line log info, this works perfectly fine. But for 2nd log, filebeat itself is creating different events for each line, so ideally filebeat should return 3 events, but it is creating 8 events. 6 events for error log which ideally should send only 1. More over i need all new line log message as msg field.
My questions
- Where i need to make changes in order to get only 3 events, logstash or filebeat?
- What changes need to do in order to achieve this.
Thanks,
Sumit