New to ELK - looking for Grok filter docs and help

Elastic Stack Logstash
1

Hello,

I am new to ELK and I'm having issues with Grok. I'm trying to parse logs that look similar to this:

"message" => "<188>362233: Jan 3 15:17:27: %IGMP_QUERIER-4-ZERO_SRC_IP_ADDR: An IGMP General Query packet with source IP address that contained all zeroes is received in VLAN 1 on port Gi2/0/52.",

This is from a Cisco switch. I would like to pull the <188> and 362233 as well as the date and time out of the message but I'm not sure how to do it. I tried Grok and I think I made it match (how can I confirm?) the <188> part using %BASE10NUM:syslog_pri filter but it was still in the message part even though it did make a new tag called syslog_pri.

Also, where can I find documentation on what the built in regex filters actually do? I'm not very good with regex so I need a little explanation (I did find the actual expressions on github).

Thank you for the help!

2

What do you have so far?

3

Well, I’ve changed it a lot but right now I’ve started over and have this:

input {
tcp {
port => 514

}
udp {
port => 514

}
}

filter {
grok {
match => ["message", "%{BASE10NUM}:syslog_pri}: %{GREEDYDATA}"]
}
syslog_pri {
}

}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

Right now I’m just trying to parse the first part.

This is giving my a grok failure though but it worked earlier so I’m not sure why.

4

And what's the result of the stdout { codec => rubydebug } output?

Please always format configuration snippets and logs as preformatted text.

5

Your match-pattern should look something like
<%{BASE10NUM:syslog_pri}>%{INT:test}: %{SYSLOGTIMESTAMP}: %{GREEDYDATA}

To test your grok I recommend Grok Debugger, you can insert your log line in the top field and your grok-patterns in the bottom.

All builtin grok-patterns can be found in the documentation:

6

Your match-pattern should look something like

This is true, although since there's no implicit ^ anchor in the grok filter I'd expect his existing expression to match anyway.

7

And what's the result of the stdout { codec => rubydebug } output?

     "message" => "<188>362403: Jan  3 18:06:12: %IGMP_QUERIER-4-ZERO_SRC_IP_ADDR: An IGMP General Query packet with source IP address that contained all zeroes is received in VLAN 1 on port Gi2/0/52.",
            "@version" => "1",
          "@timestamp" => "2017-01-03T23:06:13.834Z",
                "host" => "10.93.1.2",
                "tags" => [
    [0] "_grokparsefailure"
],
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
     "syslog_facility" => "user-level",
     "syslog_severity" => "notice"

}

8

Your match-pattern should look something like
<%{BASE10NUM:syslog_pri}>%{INT:test}: %{SYSLOGTIMESTAMP}: %{GREEDYDATA}

Thank you, I will try that out and let you know how it goes. The whole thing should still be in quotes right?

9

Nick,

Thank you, that worked!

My last question is now that I've got the tags working now do I remove the content from the message?

Would I use mutate for this? Also, what is the purpose of matching data but not tagging it? What I mean is what does adding %{SYSLOGTIMESTAMP}: %{GREEDYDATA} without the :tag part do?

Thanks again for the help guys.``

10

Ok, so I tried mutate but I can't figure out how to make it match what I want to remove.

Here is my current config:

        input {
          tcp {
            port => 514

      }
      udp {
        port => 514

      }
    }

    filter {
    grok {
    match   => ["message",  "<%{BASE10NUM:syslog_pri}>%{INT:seq}: %{SYSLOGTIMESTAMP}: %{GREEDYDATA}"]
    }
    syslog_pri {
    }
    mutate {
    remove_field => [ "message_%{BASE10NUM:syslog_pri}"]
    }
    }
    output {
      elasticsearch { hosts => ["localhost:9200"] }
      stdout { codec => rubydebug }
    }

This doesn't match anything.

How do I tell it it's part of the message but only remove a certain part? I got this far by using the docs but I'm apparently doing something wrong.

11

Capture the stuff you want to keep back into the message field. Remember to set the overwrite option.

grok {
  match => ["message", "... %{GREEDYDATA:message}"]
  overwrite => ["message"]
}
12

That did the trick.

Thank you!

So this wouldn't be possible with mutate then?

I'm just trying to figure out why mutate didn't work.

13

So this wouldn't be possible with mutate then?

What you tried to do won't work but the mutate filter's gsub option would do the job (but it's more work for you).

14

Oh ok, I was just curious.

Everything is working fine now.

Thanks again for the help.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.