# Grok/logstash efficiency

**URL:** <https://discuss.elastic.co/t/grok-logstash-efficiency/144402>\
**Category:** Logstash\
**Created:** [August 14, 2018, 8:30pm UTC](https://discuss.elastic.co/t/grok-logstash-efficiency/144402 "2018-08-14T20:30:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\
**Post date:** [August 14, 2018, 8:30pm UTC](https://discuss.elastic.co/t/grok-logstash-efficiency/144402/1 "2018-08-14T20:30:52Z")

</div>

Hi Guys,

I'm wondering if the below logstash conf could be improved in anyway, on one of my logstash servers (a lower powered one) we get about 300 e/s and this server is pinned at 100% cpu and 60% memory when things are busy and I wondered if my grok patterns/other things could be improved?

> <https://gist.github.com/if-meaton/f870f108d6ee7860f2e8e85d408a32f8>

Thanks,

Michael

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 14, 2018, 8:33pm UTC](https://discuss.elastic.co/t/grok-logstash-efficiency/144402/2 "2018-08-14T20:33:59Z")

</div>

Anchoring your grok patterns to start of line using ^ would help. Read [this](https://www.elastic.co/blog/do-you-grok-grok) post.

---

<div class="post-metadata">

**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\
**Post date:** [August 14, 2018, 9:08pm UTC](https://discuss.elastic.co/t/grok-logstash-efficiency/144402/3 "2018-08-14T21:08:32Z")

</div>

Thanks for that. I've added the start/end anchors and that seems to have sped things up although cpu is still pinned at 100%. Any other ideas on performance tuning?

Thanks,

Michael

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 14, 2018, 9:16pm UTC](https://discuss.elastic.co/t/grok-logstash-efficiency/144402/4 "2018-08-14T21:16:40Z")

</div>

```
        match => {
        "message" => [
           "%{GREEDYDATA:time} \[%{DATA:platform}\] \[%{DATA:app}\] \[%{DATA:cid}\] \[%{DATA:tid}\] \[%{DATA:loglevel}\] \[%{DATA:logsource}\] - Sequence:%{DATA:sequence}, Code:%{DATA:code}, ID:%{DATA:id}, AID:%{DATA:aid}, nToken:%{DATA:ntoken}, Method:%{DATA:method}, ExecutionTime:%{DATA:executiontime}ms, Response:%{GREEDYDATA:logmessage}",
           "%{GREEDYDATA:time} \[%{DATA:platform}\] \[%{DATA:app}\] \[%{DATA:cid}\] \[%{DATA:tid}\] \[%{DATA:loglevel}\] \[%{DATA:logsource}\] - Response from \[%{IPORHOST:ip1}, %{IPORHOST:ip2}\] to \[%{URI:url}\] ExecutionTime:%{NUMBER:executiontime}ms(%{GREEDYDATA:logmessage})?",
           "%{GREEDYDATA:time} \[%{DATA:platform}\] \[%{DATA:app}\] \[%{DATA:cid}\] \[%{DATA:tid}\] \[%{DATA:loglevel}\] \[%{DATA:logsource}\] - Code:%{DATA:code}, ID:%{DATA:id}, rID:%{DATA:rid}, Channel:%{DATA:channel}, ID:%{DATA:aid}. %{GREEDYDATA:logmessage}",
           "%{GREEDYDATA:time} \[%{DATA:platform}\] \[%{DATA:app}\] \[%{DATA:cid}\] \[%{DATA:tid}\] \[%{DATA:loglevel}\] \[%{DATA:logsource}\] - Code:%{DATA:code}, ID:%{DATA:id}, rID:%{DATA:rid}, ID:%{DATA:aid}. %{GREEDYDATA:logmessage}",
           "%{GREEDYDATA:time} \[%{DATA:platform}\] \[%{DATA:app}\] \[%{DATA:cid}\] \[%{DATA:tid}\] \[%{DATA:loglevel}\] \[%{DATA:logsource}\] - %{GREEDYDATA:logmessage}"

           ]
         }

```

Even if you anchor this, it still starts with a GREEDYDATA, so it will be very costly, because it will do a lot of back tracking. You have all those square brackets to delimit the data, so I would do that using dissect. Assuming the timestamp is two space-separated fields I would do something like

```
dissect { mapping => { "message" => "%{ts} %{+ts} [%{platform}] [%{app}] [%{cid}] [%{tid}] [%{loglevel}] [%{logsource}] - %{restOfLine}" } }

```

Then use anchored groks to match restOfLine.

---

<div class="post-metadata">

**Author:** ![if-meaton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/if-meaton/32/34035_2.png) [@if-meaton](https://discuss.elastic.co/u/if-meaton)\
**Post date:** [August 14, 2018, 9:20pm UTC](https://discuss.elastic.co/t/grok-logstash-efficiency/144402/5 "2018-08-14T21:20:00Z")

</div>

That particular pattern was given to me by the Elastic consultant that Elastic sent! But what you're saying makes sense. - I've added the anchors and i'm already seeing a massive improvement..

 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4cb37dd99beb25f3c753e3f1d5ac995de65ea060.png)

(Before and after the break is before and after i made the change and restarted logstash)

Thank you very much for your help, i'll look at your other suggestions too.

Thanks,

Michael

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 11, 2018, 9:20pm UTC](https://discuss.elastic.co/t/grok-logstash-efficiency/144402/6 "2018-09-11T21:20:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.