Grok "?" operator

Elastic Stack Logstash
1

Hello.

I am using a grok filter on certain logs that give me a problem.
A small part of them come out with one less field than the others.
example given :

input log 1
num=10 event=attack source=mach1

input log 2
num=10 source=mach2

As of now i need to check with filebeats for ["event" in msg] but with other small diffirences in the mass ammount of logs there will be a huge if/case or w/e to make the right use of the filter plus the creation of many diffirent filters.

Conclusion: I need to make use of the "?" operator as used in regular expressions but i fail to use it right.
Basically i would need something like:
number=%{NUMBER:num} (event=%{WORD:event} )?source=%{WORD:source}

But it doesn't work like this and i can't find the right way to use it in grok.What is the right syntax for something like this so i won't need to make use of if condition in filebeat?Or actually is it even possble?
One thing i could think of would be something close to ( | event=%{}) but i aint sure how to implement it the right way.

Thanks in advance.

2

I would recommend you have a look at the kv filter, which is designed to parse lists of key-value pairs that can appear in any order to only sometimes be present.

3

Thanks for your reply.
1st if someone can give me an answer i would appreciate it a lot.I would like to learn how to do it.
2nd is the kv filter good performance wise compared to making my own regexes+patterns for everything?

4

@sancroth It seems to me that this question would better fit into the Logstash forum?

5

I think logstash/ingest node use oniguruma regex engine. The grok pattern will be compiled down to a regex pattern using named capture groups. See logstash grok docs on Custom Patterns for using a capture group directly.

6

For some reason i thought i posted it there :o Terribly sorry.If a mod can swap places it would be nice :slight_smile: (looking at you ruf :smiley: )

7

And just for the information and the help you given me,
I had to manipulate fortigate logs.I went with KV filter as suggested after clearing the first part of the logs with regexes.
Only thing i haven't figured out yet is if there is a way to change data types with KV since it automatically matches the delimeter of choice and creates the fields.
With grok you could use %{NUMBER:num:int} and BOOM.Still looking in to it.

Thanks for everyones help!Really appreciated!

8

I just moved the topic to logstash.

1 Like
9

This topic was automatically closed after 21 days. New replies are no longer allowed.