# Grok parsing dynamic log-lines

**URL:** <https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997>\
**Category:** Logstash\
**Created:** [May 23, 2018, 1:05pm UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997 "2018-05-23T13:05:01Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dante91](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@Dante91](https://discuss.elastic.co/u/Dante91)\
**Post date:** [May 23, 2018, 1:05pm UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997/1 "2018-05-23T13:05:02Z")

</div>

Hello together,  
I want to log different topics from my router, giving me dynamic logs like

> Apr/25/2018 07:19:37 ipsec,debug,packet 852310ad 5ebd8ba7 5ee02726 b0f1e739 6dafe48d 6c2f812f  
> Apr/25/2018 07:19:37 ipsec,debug encrypted payload by IV  
> Apr/25/2018 08:01:11 system,info,account user admin logged out from 192.168.88.1 via telnet

As you can see, the topic-information after the timestamp, could not be set static because depending on the topic, there could be one or more topics, all seperated through a comma.  
Because I don´t want to set one filter per possible log-topic I want you to ask, if theres any filter-possibility where I can split this log in timestamp, topic1, topic,2, topic3,... and message

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 24, 2018, 6:21am UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997/2 "2018-05-24T06:21:50Z")

</div>

Capture all topics into a single field (e.g. with a NOTSPACE pattern), then turn that field into an array with a mutate filter's split option.

---

<div class="post-metadata">

**Author:** ![Dante91](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@Dante91](https://discuss.elastic.co/u/Dante91)\
**Post date:** [May 25, 2018, 6:28am UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997/3 "2018-05-25T06:28:53Z")

</div>

Thanks for your reply, I approximately know what you mean, but I don´t know how to realize.  
How could that filter look like?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 25, 2018, 6:41am UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997/4 "2018-05-25T06:41:27Z")

</div>

The grok filter or the mutate filter?

---

<div class="post-metadata">

**Author:** ![Dante91](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@Dante91](https://discuss.elastic.co/u/Dante91)\
**Post date:** [May 25, 2018, 7:02am UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997/5 "2018-05-25T07:02:30Z")

</div>

My grok-filter looks like this:

> if [type] == "Router1" {  
> grok {  
> match =\> {"message" =\> "%{WORD:TOPIC1},%{WORD:TOPIC2},%{WORD:TOPIC3} {GREEDYDATA:MESSAGE}"}  
> }  
> }  
> }

So far it works fine as long as there are exactly three topics. Otherwise I get the \_grokparsefailure and my log is not split correctly. How should I adjust my filter-config?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 25, 2018, 7:47am UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997/6 "2018-05-25T07:47:30Z")

</div>

Make sure you match the timestamp and replace `%{WORD:TOPIC1},%{WORD:TOPIC2},%{WORD:TOPIC3}` with `%{NOTSPACE:topic}`, i.e. something like this:

```
^(... something that matches the timestamp ...) %{NOTSPACE:topic} %{GREEDYDATA:message}
```

---

<div class="post-metadata">

**Author:** ![Dante91](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@Dante91](https://discuss.elastic.co/u/Dante91)\
**Post date:** [May 25, 2018, 10:17am UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997/7 "2018-05-25T10:17:18Z")

</div>

Thanks a lot, that was the pattern I was looking for. I noticed seperating the topics wouldn´t make sense because the sequence of topics chanches. Searching with kibana is easier with the simple filter:

> filter {  
> if [type] == "MikroTik" {  
> grok {  
> match =\> {"message" =\> "%{NOTSPACE:TOPIC} %{GREEDYDATA:MESSAGE}"}  
> }  
> }  
> }

Thanks magnusbaeck

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 25, 2018, 11:17am UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997/8 "2018-05-25T11:17:16Z")

</div>

> I noticed seperating the topics wouldn´t make sense because the sequence of topics chanches.

That's why you should use a mutate filter to split the string "a,b,c" into an array (["a", "b", "c"]):

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2018, 11:17am UTC](https://discuss.elastic.co/t/grok-parsing-dynamic-log-lines/132997/9 "2018-06-22T11:17:20Z")

</div>

Closed on June 22, 2018, 11:17am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.