Grok pattern for optional field in ingest pipeline

[alfianaf (Alfian Aulia Firdaus)]

Hello, I'm working with ingest pipeline and grok processor right now.
I need to filter log message with grok processor but there's optional fields,

here's the log example

[Nest] 6448 - 04/26/2022, 3:30:03 PM [activity] message +1ms

this one has grok pattern

%{GREEDYDATA:generated_nest} - %{GREEDYDATA:timestamp} \\[%{GREEDYDATA:activity}\\] %{GREEDYDATA:message} %{GREEDYDATA:generated_ms}

while the log also has this field

[Nest] 6448 - 04/26/2022, 3:31:25 PM username | [activity] message | reference +1ms

so the grok pattern should be

%{GREEDYDATA:generated_nest} - %{GREEDYDATA:timestamp} %{GREEDYDATA:username} \\| \\[%{GREEDYDATA:activity}\\] %{GREEDYDATA:message} \\| %{GREEDYDATA:reference} %{GREEDYDATA:generated_ms}

from what you can see, the username and reference field is optional (these two fields is not depend each other),
is there any way to use one grok pattern, so I don't need to mention each pattern while using "ignore_failure" on each processor?

any help would be appreciated,

Thanks

[Rios]

Here is your grok:
\[%{GREEDYDATA:process}\] %{POSINT:procid} - %{GREEDYDATA:timestamp} \[%{GREEDYDATA:activity}\] %{GREEDYDATA:message} %{GREEDYDATA:generated_ms}

I would avoid GREEDYDATA since it's slow and add date formatting.

filter {	
    grok {
	 # [Nest] 6448 - 04/26/2022, 3:30:03 PM [activity] message +1ms
      match => { "message" => "\[%{DATA:process}\] %{POSINT:procid} - %{DATA:timestamp} \[%{DATA:activity}\] %{DATA:message} %{GREEDYDATA:generated_ms}" }
    }
	
    date {
      match => [ "timestamp", "MM/dd/yyyy, H:mm:ss a"  ]
      timezone => "Asia/Singapore"
     }
}

Result:

{
        "@version" => "1",
       "timestamp" => "04/26/2022, 3:30:03 PM",
        "activity" => "activity",
    "generated_ms" => "+1ms",
      "@timestamp" => 2022-04-25T19:30:03.000Z,
         "process" => "Nest",
          "procid" => "6448",
         "message" => "[Nest] 6448 - 04/26/2022, 3:30:03 PM [activity] message +1ms"

    ]
}

[alfianaf (Alfian Aulia Firdaus)]

hello, thanks for the response,
but it seems like you missed my point, I have two almost similar log, there's username field and reference field which is possible to be included into the log.
the grok pattern you provided won't matched with my 2nd log,
is there any way to have one grok pattern with those two different log?

[Rios]

You have to handle optional fields or have 2 match patterns. This is 1st option with handling date format.

    grok {
	  patterns_dir => "./pattern"
      match => { "message" => "\[%{DATA:process}\] %{POSINT:procid} - %{CDATE:timestamp} (%{DATA:username}\s\|\s)?\[%{DATA:activity}\] %{DATA:message} (\|\s%{DATA:ref}\s)?%{GREEDYDATA:generated_ms}" }
    }

Put a file in the pattern directory with content:
CDATE %{DATE_US}\, %{TIME} %{WORD}