Grok-Filter not working

Elastic Stack Logstash
1

Hey,
I want to log my MikroTik-Router. Thatfor I get for example the following log:

May/03/2018 17:37:01 system,info,account user admin logged out from 192.168.88.1 via telnet

With help from the grok debugger at grokdebug.herokuapp.com I get the following config:

input {
tcp {
port => 5514
type => "MikroTik"
}
udp {
port => 5514
type => "MikroTik"
}
}
filter {
if [type] == "MikroTik" {
grok {
match => {"message" => "(?"<"logtimestamp">"%{MONTH}/%{MONTHDAY}/%{YEAR} %{TIME}) %{WORD:source},%{WORD:level},%{GREEDYDATA:message}"}
}
mutate {
add_tag => ["grokked"]
add_field => {
"source" => "%{source)"
"level" => "%(level)"
}
}
}
}
output {
stdout { codec => rubydebug }
}

I had to do the brackets around the "logtimestamp" because of the formatting. When I start logstash with this config I get following output:

{
"host" => "192.168.88.3",
"message" => "system,info,account user admin logged out from 192.168.88.1 via telnet",
"level" => "%(level)",
"source" => "%{source)",
"@version" => "1",
"type" => "MikroTik",
"@timestamp" => 2018-05-08T02:44:38.508Z,
"tags" => [
[0] "_grokparsefailure",
[1] "grokked"
]
}

Unfortunately the grok-filter doesn´t work for me. My logstash-version is the 6.2.3
Hope someone can help me debugging my config.

2

Your configuration isn't consistent with the output you say you're getting. If the grok filter indeed wasn't working, how come the timestamp is stripped away from the message field?

3

Hey magnusbaeck,
thanks for the quick reply. Where could this timestamp come from?
I start logstash with my testconfig.conf and following parameters:

./logstash -f testconfig.conf --path.settings /etc/logstash/

The conf.d-folder in /etc/logstash is empty and my logstash folder is in /usr/share/logstash/bin.

4 
grok { match => {"message" => "(?<logtimestamp>%{MONTH}/%{MONTHDAY}/%{YEAR} %{TIME}) %{WORD:source},%{WORD:level},%{GREEDYDATA:message}"} }

will match that message. What are you trying to do with the mutate+add_field?

"source" => "%{source)"

You have mis-matched braces and parentheses. If we correct that, the source field already exists, so adding a second source field just forces it to be an array. Similarly for grokking a message field using a regexp that has an item called message -- message ends up as an array, which is probably not what you want.

         "message" => [
        [0] "May/03/2018 17:37:01 system,info,account user admin logged out from 192.168.88.1 via telnet",
        [1] "account user admin logged out from 192.168.88.1 via telnet"
    ],
5

Hey Badger,
to eliminate configuration misstakes, I installed a completely new machine with the actual versions of elk-stack.
In my filter-options I only placed the grok-filter without any other options and renamed the fields:

filter {
if [type] == "MikroTik" {
grok {
match => {"test012" => "(?%{MONTH}/%{MONTHDAY}/%{YEAR} %{TIME}) %{WORD:test123},%{GREEDYDATA:test234}"}
}
}
}

So there should be no field that is already "in use" by logstash. Unfortunately I don´t even get the hoped result:

{
"tags" => [
[0] "_grokparsefailure"
],
"message" => "system,info,account user admin logged in from 192.168.88.1 via telnet",
"@version" => "1",
"type" => "MikroTik",
"host" => "192.168.88.3",
"@timestamp" => 2018-05-09T08:32:20.308Z
}

Again there is this grokparsefailure. Isn´t there any log that could explain where this error comes from?

6

The grok filter fails because your event doesn't have a test012 field.

7

Thanks for the quick reply, even if I change the test012-fielt into "message" the output doesn´t change.

{
"@version" => "1",
"@timestamp" => 2018-05-09T09:40:53.305Z,
"host" => "192.168.88.3",
"tags" => [
[0] "_grokparsefailure"
],
"type" => "MikroTik",
"message" => "system,info,account user admin logged in from 192.168.88.1 via telnet"
}

Again the "_grokparsefailure"-tag

8

message obviously doesn't match the grok expression.

9

Then I think that I understood something wrong. In my output there are standard defined variables like "@version", "host", "message", ....
The type-tag was given by logstash on the input-conf. Now I want the grok filter to seperate this field "message" into seperate fields, in my case "test123" and "test234".
What did I understand wrong?

10

In the grok filter you said it should match a date, followed by a word, followed by the rest of the line. But your message does not have a date, so grok is going to fail.

11

Hey badger,
that was the solution, I didn´t notice that the timestamp wasn´t already in my message output. Thanks to you and magnusbaeck.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.