How to provide default values in case of grok parse failure?

Elastic Stack Logstash
1

I am using grok for parsing logs.My requirement is to set default value for any missing string field as null and for any integer field as 0.

Sample logs are as follows:--

Jan 23 12:16:52 [10.10.10.10] <13> 10.11.12.13  10.20.30.40 loc=1810756

The grok filter I am using is:--

%{SYSLOGTIMESTAMP:ts} \[%{IPV4:f1}\] \<%{USER:hField1}\> %{IPV4:hIp1} *%{IPV4:hIp2} loc=%{INT:loc}

For example,if "f1" field is missing then in that case f1 should be set as null and if loc is missing then it must be set as 0.

2

You can use a mutate filter with add_field to add the fields with their default values prior to the grok filter, then use the grok filter's overwrite option to tell grok that those fields are okay to overwrite.

3

But if the value(e.g. f1) is missing then the grok parser will fail and no subsequent fields will be parsed.
And also, is there any way to specify all the fields in strip_field of mutate filter by a single keyword like *.

4

But if the value(e.g. f1) is missing then the grok parser will fail and no subsequent fields will be parsed.

Yes...? If the value is missing then the old default value will still be there.

And also, is there any way to specify all the fields in strip_field of mutate filter by a single keyword like *.

No.

5

What I mean to say is if f1 is missing then grok parser will fail and then I will not be able to parse either of subsequent fields like hField1 or hIp1 ,etc which might not be missing from the logs and can be parsed from it.

6

Okay, but that's a different problem. You can either make parts of the grok expression optional with (...)? or you can list multiple expressions in the same grok filter. They will be tried in order and the first match wins. There's an example in the documentation.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.