Multile pattren in single Grok Filter

[Tek_Chand]

Hello All,

I have written multiple pattern in single grok filter, but my logs are parse only first filter. Below are the filter in my Grok

 match => { "message" => [ "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}",
                          "\I\,\s\[(?<date-time>[\d\-\w\:\.]+)\s\#(?<pid>[\d]+)\]\s\s(?<loglevel>[\w]+)\s\--\s\:\s\[(?<request-id>[\d\-\w]+)\]\s(?<path>[\w\s\"\/\.]+)\s(?<message>.*)",
                          "\I\,\s\[(?<date-time>[\d\-\w\:\.]+)\s\#(?<pid>[\d]+)\]\s\s(?<loglevel>[\w]+)\s\--\s\:\s\[(?<request-id>[\d\-\w]+)\]\s(?<message>.*)",
                          "\I\,\s\[(?<date-time>[\d\-\w\:\.]+)\s\#(?<pid>[\d]+)\]\s\s(?<loglevel>[\w]+)\s\--\s\:\s\[(?<request-id>[\d\-\w]+)\]\s(?<device-id>[\w\s\:]+)" ] }

Any suggestion will be appreciated.

Thanks in advance.

[Shaoranlaos (Christian Stockhaus)]

could you post some sample lines?

Not sure if i read it correctly but are these pattern for different log lines or each for the same and they extract different fields?

if these filters should all be executed for the same log line you will need to disable the break_on_match option of the grok filter:
https://www.elastic.co/guide/en/logstash/6.2/plugins-filters-grok.html#plugins-filters-grok-break_on_match

[Tek_Chand]

Hello Shaoranlaos,

Above patterns are for different log lines. But some filelds are common in all logs. So my patterns are look generic. Please find the some sample logs:

I, [2018-07-23T08:49:09.854771 #22418]  INFO -- : [81ee6e72-e1b3-4480-b664-0e215cdf23a8] Started POST "/api/v1/locations.json" for 127.0.0.1 at 2018-07-23 08:49:09 +0000
I, [2018-07-23T08:49:09.856826 #22418]  INFO -- : [81ee6e72-e1b3-4480-b664-0e215cdf23a8] Processing by Api::V1::LocationsController#create as JSON
I, [2018-07-23T08:49:09.860378 #22418]  INFO -- : [81ee6e72-e1b3-4480-b664-0e215cdf23a8] Current device: 1380 : 001
I, [2018-07-23T08:49:09.861850 #22418]  INFO -- : [81ee6e72-e1b3-4480-b664-0e215cdf23a8] Device :: 1380 :: 5

Please help to fix this issue or let me know if i need to made any changes in my patterns.

Thanks.

[Shaoranlaos (Christian Stockhaus)]

in my tries it is the second pattern that will always match and not the first (because your log lines are not syslogs)

anyway could you try to change the order of the patterns and be more specific in the matching because all 3 pattern match the provided lines and extract because of this for the specific fields nonsense
it needs some identifing word or character to correctly differentiate between the patterns

for this you also could try to only use the general pattern (where only message is extracted, second pattern in your starting post) and then make the specific field extraction in an extra grok filter on the message field(would probaly reduce some overhead and maintenance cost).

e.g. (like i think it will make sense)

 match => { "message" => [ "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}",
                          "\I\,\s\[(?<date-time>[\d\-\w\:\.]+)\s\#(?<pid>[\d]+)\]\s\s(?<loglevel>[\w]+)\s\--\s\:\s\[(?<request-id>[\d\-\w]+)\]\sStarted\s%{WORD:verb}\s\"(?<path>[\w\s\"\/\.]+)\"\s(?<message>.*)",
                          "\I\,\s\[(?<date-time>[\d\-\w\:\.]+)\s\#(?<pid>[\d]+)\]\s\s(?<loglevel>[\w]+)\s\--\s\:\s\[(?<request-id>[\d\-\w]+)\]\s(?:Current\s)?[dD]evice[\s:]+(?<device-id>[\w\s\:]+)",
                          "\I\,\s\[(?<date-time>[\d\-\w\:\.]+)\s\#(?<pid>[\d]+)\]\s\s(?<loglevel>[\w]+)\s\--\s\:\s\[(?<request-id>[\d\-\w]+)\]\s(?<message>.*)" ] }

this is only a quick shoot from my side i will see if i find the time today to take a closer look

[Tek_Chand]

Hello Shaoranlaos,

Thank you for your help. You are right that was my second pattern which match my logs. I wil also look into your approach and try to be more specific.

Thanks once again.