# Grok filter. How do I break up the message information?

**URL:** <https://discuss.elastic.co/t/grok-filter-how-do-i-break-up-the-message-information/110566>\
**Category:** Logstash\
**Created:** [December 6, 2017, 7:11pm UTC](https://discuss.elastic.co/t/grok-filter-how-do-i-break-up-the-message-information/110566 "2017-12-06T19:11:01Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![E\_Marshall](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e_marshall/32/14191_2.png) [@E\_Marshall](https://discuss.elastic.co/u/E_Marshall)\
**Post date:** [December 6, 2017, 7:11pm UTC](https://discuss.elastic.co/t/grok-filter-how-do-i-break-up-the-message-information/110566/1 "2017-12-06T19:11:01Z")

</div>

I created a filter to break apart our log files and am having the following issue. I'm not able to figure out how to save the parts of the "message" to their own field or tag or whatever you call it. I'm 3 days new to logstash and have had zero luck with finding someone here who knows it.

So for an example lets say this is your log line in a log file  
_2017-12-05 [user:edjm1971] msg:This is a message from the system._

And what you want to do is to get the value of the user and set that into some index mapping so you can search for all logs that were by that user. Also, you should see the information from the message in their own fields in Kibana.

My pipeline.conf file for logstash is like  
grok {  
match =\> {  
"message" =\> "%{TIMESTAMP\_ISO8601:timestamp} [sid:%{USERNAME:sid} msg:%{DATA:message}"  
}  
add\_tag =\> ["foo\_tag", "some\_user\_value\_from\_sid\_above"]  
}

Now when I run the logger to create logs data gets over to ES and I can see the data in KIBANA but I don't see foo\_tag at all with the sid value.  
How exactly do I use this to create the new tag that gets stored into ES so I can see the data I want from the message?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 18, 2017, 7:17pm UTC](https://discuss.elastic.co/t/grok-filter-how-do-i-break-up-the-message-information/110566/2 "2017-12-18T19:17:01Z")

</div>

You don't need `add_tag`. In the grok expression itself you're asking for the components of the string to be extracted to the fields `timestamp`, `sid`, and `message`.

To overwrite the existing `message` field with the one extracted in this grok filter you'll have to use the grok filter's `overwrite` option.

---

<div class="post-metadata">

**Author:** ![E\_Marshall](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e_marshall/32/14191_2.png) [@E\_Marshall](https://discuss.elastic.co/u/E_Marshall)\
**Post date:** [December 20, 2017, 5:44pm UTC](https://discuss.elastic.co/t/grok-filter-how-do-i-break-up-the-message-information/110566/3 "2017-12-20T17:44:21Z")

</div>

Thank you. I managed to get this all figured out and working. One thing though was that when I put some custom patterns in (which I had tested and were valid) into an external file in the pattern directory it was painfully slow. When I just opted to use a pre built GROK pattern that could also do what I wanted it was very fast.  
Is it normal for external pattern files to run slowly?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 21, 2017, 6:35am UTC](https://discuss.elastic.co/t/grok-filter-how-do-i-break-up-the-message-information/110566/4 "2017-12-21T06:35:29Z")

</div>

The location of the patterns doesn't matter but the quality of them does. Frequent use of DATA and GREEDYDATA can have extreme effects on the execution time.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 18, 2018, 6:35am UTC](https://discuss.elastic.co/t/grok-filter-how-do-i-break-up-the-message-information/110566/5 "2018-01-18T06:35:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.