Logstash parsing each log multiple times

This is how my logs look like. I am using multi grok parser to parse different type of logs but in same input file. Below is the grok filter that works for first log

^%{SOCBOX_HEADER}%{NUMBER}:%{NUMBER}:%{NUMBER} %{USERNAME:Server} %{USERNAME:AuditResult} %{NUMBER:EventID} (?<Message>([^#]*))Subject:Security ID:%{USERNAME:security_id}Account Name:%{USERNAME:account_name}Account Domain:%{USERNAME:account_domain}Logon ID:%{USERNAME:logon_id}Logon Type:%{NUMBER:logon_type}New Logon:Security ID:%{USERNAME:security_id2}Account Name:%{DATA:account_name2}Account Domain:%{USERNAME:account_domain2}Logon ID:%{USERNAME:logon_id2}Logon GUID:\{%{USERNAME:logon_guid}\}Process Information:Process ID:%{USERNAME:process_id}Process Name:%{USERNAME:process_name}Network Information:Workstation Name:Source Network Address:%{USERNAME:source_network_address}Source Port:%{USERNAME:source_port}Detailed Authentication Information:Logon Process:%{USERNAME:logon_process}Authentication Package:%{USERNAME:authentication_package}Transited Services:%{USERNAME:transited_service}Package Name \(NTLM only\):%{USERNAME:package_name}Key Length:%{INT:key_length}(?<Message2>([^#]*))

and below the gork filter which works for second log

^%{SOCBOX_HEADER}%{NUMBER}:%{NUMBER}:%{NUMBER} %{USERNAME:Server} %{USERNAME:AuditResult} %{NUMBER:EventID} (?<Message>([^#]*))Subject:Security ID:%{USERNAME:security_id}Account Name:%{DATA:account_name}Account Domain:%{DATA:account_domain}Logon ID:%{USERNAME:logon_id}(Logon Type:%{NUMBER:logon_type})?(?<Message2>([^#]*))

Issue is that some of the fields of the second grok filter do match with the first one and because of that the logstash is parsing each log twice and in some cases it is even thrice. I have tried setting break_on_match => true or even tried making field optional like (?:%{IP:ip})? or %{IP:ip}? or (?:\s+%{IP:ip}) but nothing works. There is not issue with the grok filter, no warnings or error logs expect each field gets parsed multiple times and in kibana I can see same output twice with comma in between them like > |syslog_src|

> |August 16th 2018, 17:00:00.000, August 16th 2018, 17:00:00.000, August 16th 2018, 17:00:00.000|
>     |t  syslog_ts|       |2018-08-17 00:26:16, 2018-08-17 00:26:16, 2018-08-17 00:26:16|

It's hard to give specific advice without knowing what your configuration looks like, but you'll probably want to use a single grok filter containing multiple grok expressions. Logstash will try them in order and stop as soon as there's a match.

Here is link to my config file https://pastebin.com/0N3igFX8

As I suspected. Use a single grok filter that lists multiple expressions (the documentation contains a syntax example). The break_on_match option you're using has no effect when only a single expression is specified.

Thank you very much sir, you solved all my issues.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.