Grok parsing dynamic log-lines

Elastic Stack Logstash
1

Hello together,
I want to log different topics from my router, giving me dynamic logs like

Apr/25/2018 07:19:37 ipsec,debug,packet 852310ad 5ebd8ba7 5ee02726 b0f1e739 6dafe48d 6c2f812f
Apr/25/2018 07:19:37 ipsec,debug encrypted payload by IV
Apr/25/2018 08:01:11 system,info,account user admin logged out from 192.168.88.1 via telnet

As you can see, the topic-information after the timestamp, could not be set static because depending on the topic, there could be one or more topics, all seperated through a comma.
Because I don´t want to set one filter per possible log-topic I want you to ask, if theres any filter-possibility where I can split this log in timestamp, topic1, topic,2, topic3,... and message

2

Capture all topics into a single field (e.g. with a NOTSPACE pattern), then turn that field into an array with a mutate filter's split option.

1 Like
3

Thanks for your reply, I approximately know what you mean, but I don´t know how to realize.
How could that filter look like?

4

The grok filter or the mutate filter?

5

My grok-filter looks like this:

if [type] == "Router1" {
grok {
match => {"message" => "%{WORD:TOPIC1},%{WORD:TOPIC2},%{WORD:TOPIC3} {GREEDYDATA:MESSAGE}"}
}
}
}

So far it works fine as long as there are exactly three topics. Otherwise I get the _grokparsefailure and my log is not split correctly. How should I adjust my filter-config?

6

Make sure you match the timestamp and replace %{WORD:TOPIC1},%{WORD:TOPIC2},%{WORD:TOPIC3} with %{NOTSPACE:topic}, i.e. something like this:

^(... something that matches the timestamp ...) %{NOTSPACE:topic} %{GREEDYDATA:message}
7

Thanks a lot, that was the pattern I was looking for. I noticed seperating the topics wouldn´t make sense because the sequence of topics chanches. Searching with kibana is easier with the simple filter:

filter {
if [type] == "MikroTik" {
grok {
match => {"message" => "%{NOTSPACE:TOPIC} %{GREEDYDATA:MESSAGE}"}
}
}
}

Thanks magnusbaeck

8

I noticed seperating the topics wouldn´t make sense because the sequence of topics chanches.

That's why you should use a mutate filter to split the string "a,b,c" into an array (["a", "b", "c"]):

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.