This is how my logs look like. I am using multi grok parser to parse different type of logs but in same input file. Below is the grok filter that works for first log
^%{SOCBOX_HEADER}%{NUMBER}:%{NUMBER}:%{NUMBER} %{USERNAME:Server} %{USERNAME:AuditResult} %{NUMBER:EventID} (?<Message>([^#]*))Subject:Security ID:%{USERNAME:security_id}Account Name:%{USERNAME:account_name}Account Domain:%{USERNAME:account_domain}Logon ID:%{USERNAME:logon_id}Logon Type:%{NUMBER:logon_type}New Logon:Security ID:%{USERNAME:security_id2}Account Name:%{DATA:account_name2}Account Domain:%{USERNAME:account_domain2}Logon ID:%{USERNAME:logon_id2}Logon GUID:\{%{USERNAME:logon_guid}\}Process Information:Process ID:%{USERNAME:process_id}Process Name:%{USERNAME:process_name}Network Information:Workstation Name:Source Network Address:%{USERNAME:source_network_address}Source Port:%{USERNAME:source_port}Detailed Authentication Information:Logon Process:%{USERNAME:logon_process}Authentication Package:%{USERNAME:authentication_package}Transited Services:%{USERNAME:transited_service}Package Name \(NTLM only\):%{USERNAME:package_name}Key Length:%{INT:key_length}(?<Message2>([^#]*))
and below the gork filter which works for second log
^%{SOCBOX_HEADER}%{NUMBER}:%{NUMBER}:%{NUMBER} %{USERNAME:Server} %{USERNAME:AuditResult} %{NUMBER:EventID} (?<Message>([^#]*))Subject:Security ID:%{USERNAME:security_id}Account Name:%{DATA:account_name}Account Domain:%{DATA:account_domain}Logon ID:%{USERNAME:logon_id}(Logon Type:%{NUMBER:logon_type})?(?<Message2>([^#]*))
Issue is that some of the fields of the second grok filter do match with the first one and because of that the logstash is parsing each log twice and in some cases it is even thrice. I have tried setting break_on_match => true or even tried making field optional like (?:%{IP:ip})? or %{IP:ip}? or (?:\s+%{IP:ip}) but nothing works. There is not issue with the grok filter, no warnings or error logs expect each field gets parsed multiple times and in kibana I can see same output twice with comma in between them like > |syslog_src|
> |August 16th 2018, 17:00:00.000, August 16th 2018, 17:00:00.000, August 16th 2018, 17:00:00.000|
> |t syslog_ts| |2018-08-17 00:26:16, 2018-08-17 00:26:16, 2018-08-17 00:26:16|