# New to ELK - looking for Grok filter docs and help

**URL:** <https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480>\
**Category:** Logstash\
**Created:** [January 3, 2017, 8:37pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480 "2017-01-03T20:37:05Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![dotson83](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@dotson83](https://discuss.elastic.co/u/dotson83)\
**Post date:** [January 3, 2017, 8:37pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/1 "2017-01-03T20:37:06Z")

</div>

Hello,

I am new to ELK and I'm having issues with Grok. I'm trying to parse logs that look similar to this:

"message" =\> "\<188\>362233: Jan 3 15:17:27: %IGMP\_QUERIER-4-ZERO\_SRC\_IP\_ADDR: An IGMP General Query packet with source IP address that contained all zeroes is received in VLAN 1 on port Gi2/0/52.",

This is from a Cisco switch. I would like to pull the \<188\> and 362233 as well as the date and time out of the message but I'm not sure how to do it. I tried Grok and I think I made it match (how can I confirm?) the \<188\> part using %BASE10NUM:syslog\_pri filter but it was still in the message part even though it did make a new tag called syslog\_pri.

Also, where can I find documentation on what the built in regex filters actually do? I'm not very good with regex so I need a little explanation (I did find the actual expressions on github).

Thank you for the help!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 3, 2017, 10:54pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/2 "2017-01-03T22:54:00Z")

</div>

What do you have so far?

---

<div class="post-metadata">

**Author:** ![dotson83](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@dotson83](https://discuss.elastic.co/u/dotson83)\
**Post date:** [January 3, 2017, 11:09pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/3 "2017-01-03T23:09:18Z")

</div>

Well, I’ve changed it a lot but right now I’ve started over and have this:

input {  
tcp {  
port =\> 514

}  
udp {  
port =\> 514

}  
}

filter {  
grok {  
match =\> ["message", "%{BASE10NUM}:syslog\_pri}: %{GREEDYDATA}"]  
}  
syslog\_pri {  
}

}  
output {  
elasticsearch { hosts =\> ["localhost:9200"] }  
stdout { codec =\> rubydebug }  
}

Right now I’m just trying to parse the first part.

This is giving my a grok failure though but it worked earlier so I’m not sure why.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 4, 2017, 7:28am UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/4 "2017-01-04T07:28:18Z")

</div>

And what's the result of the `stdout { codec => rubydebug }` output?

Please always format configuration snippets and logs as preformatted text.

---

<div class="post-metadata">

**Author:** ![nick.e](https://avatars.discourse-cdn.com/v4/letter/n/8dc957/32.png) [@nick.e](https://discuss.elastic.co/u/nick.e)\
**Post date:** [January 4, 2017, 8:06am UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/5 "2017-01-04T08:06:17Z")

</div>

Your match-pattern should look something like  
`<%{BASE10NUM:syslog_pri}>%{INT:test}: %{SYSLOGTIMESTAMP}: %{GREEDYDATA}`

To test your grok I recommend [Grok Debugger](https://grokdebug.herokuapp.com/), you can insert your log line in the top field and your grok-patterns in the bottom.

All builtin grok-patterns can be found in the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html):

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 4, 2017, 8:14am UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/6 "2017-01-04T08:14:03Z")

</div>

> Your match-pattern should look something like

This is true, although since there's no implicit `^` anchor in the grok filter I'd expect his existing expression to match anyway.

---

<div class="post-metadata">

**Author:** ![dotson83](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@dotson83](https://discuss.elastic.co/u/dotson83)\
**Post date:** [January 4, 2017, 3:02pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/7 "2017-01-04T15:02:52Z")

</div>

> And what's the result of the stdout { codec =\> rubydebug } output?

```
     "message" => "<188>362403: Jan 3 18:06:12: %IGMP_QUERIER-4-ZERO_SRC_IP_ADDR: An IGMP General Query packet with source IP address that contained all zeroes is received in VLAN 1 on port Gi2/0/52.",
            "@version" => "1",
          "@timestamp" => "2017-01-03T23:06:13.834Z",
                "host" => "10.93.1.2",
                "tags" => [
    [0] "_grokparsefailure"
],
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
     "syslog_facility" => "user-level",
     "syslog_severity" => "notice"

```

}

---

<div class="post-metadata">

**Author:** ![dotson83](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@dotson83](https://discuss.elastic.co/u/dotson83)\
**Post date:** [January 4, 2017, 3:08pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/8 "2017-01-04T15:08:45Z")

</div>

> Your match-pattern should look something like  
> \<%{BASE10NUM:syslog\_pri}\>%{INT:test}: %{SYSLOGTIMESTAMP}: %{GREEDYDATA}

Thank you, I will try that out and let you know how it goes. The whole thing should still be in quotes right?

---

<div class="post-metadata">

**Author:** ![dotson83](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@dotson83](https://discuss.elastic.co/u/dotson83)\
**Post date:** [January 4, 2017, 3:48pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/9 "2017-01-04T15:48:17Z")

</div>

Nick,

Thank you, that worked!

My last question is now that I've got the tags working now do I remove the content from the message?

Would I use mutate for this? Also, what is the purpose of matching data but not tagging it? What I mean is what does adding %{SYSLOGTIMESTAMP}: %{GREEDYDATA} without the :tag part do?

Thanks again for the help guys.``

---

<div class="post-metadata">

**Author:** ![dotson83](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@dotson83](https://discuss.elastic.co/u/dotson83)\
**Post date:** [January 4, 2017, 6:19pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/10 "2017-01-04T18:19:03Z")

</div>

Ok, so I tried mutate but I can't figure out how to make it match what I want to remove.

Here is my current config:

> ```
> input {
> tcp {
> port => 514
> 
> ```

> ```
> }
> udp {
> port => 514
> 
> ```

> ```
> }
> }
> 
> ```

> ```
> filter {
> grok {
> match => ["message", "<%{BASE10NUM:syslog_pri}>%{INT:seq}: %{SYSLOGTIMESTAMP}: %{GREEDYDATA}"]
> }
> syslog_pri {
> }
> mutate {
> remove_field => ["message_%{BASE10NUM:syslog_pri}"]
> }
> }
> output {
> elasticsearch { hosts => ["localhost:9200"] }
> stdout { codec => rubydebug }
> }
> 
> ```

This doesn't match anything.

How do I tell it it's part of the message but only remove a certain part? I got this far by using the docs but I'm apparently doing something wrong.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 4, 2017, 6:53pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/11 "2017-01-04T18:53:35Z")

</div>

Capture the stuff you want to keep back into the `message` field. Remember to set the `overwrite` option.

```nohighlight
grok {
  match => ["message", "... %{GREEDYDATA:message}"]
  overwrite => ["message"]
}

```

---

<div class="post-metadata">

**Author:** ![dotson83](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@dotson83](https://discuss.elastic.co/u/dotson83)\
**Post date:** [January 4, 2017, 8:58pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/12 "2017-01-04T20:58:43Z")

</div>

That did the trick.

Thank you!

So this wouldn't be possible with mutate then?

I'm just trying to figure out why mutate didn't work.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 8, 2017, 4:46pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/13 "2017-01-08T16:46:55Z")

</div>

> So this wouldn't be possible with mutate then?

What you tried to do won't work but the mutate filter's gsub option would do the job (but it's more work for you).

---

<div class="post-metadata">

**Author:** ![dotson83](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@dotson83](https://discuss.elastic.co/u/dotson83)\
**Post date:** [January 8, 2017, 6:09pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/14 "2017-01-08T18:09:09Z")

</div>

Oh ok, I was just curious.

Everything is working fine now.

Thanks again for the help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2017, 6:09pm UTC](https://discuss.elastic.co/t/new-to-elk-looking-for-grok-filter-docs-and-help/70480/15 "2017-02-05T18:09:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.