# Grok increase CPU usage to 700+%

**URL:** <https://discuss.elastic.co/t/grok-increase-cpu-usage-to-700/256633>\
**Category:** Logstash\
**Created:** [November 25, 2020, 10:15am UTC](https://discuss.elastic.co/t/grok-increase-cpu-usage-to-700/256633 "2020-11-25T10:15:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![vaclav1](https://avatars.discourse-cdn.com/v4/letter/v/c68b51/32.png) [@vaclav1](https://discuss.elastic.co/u/vaclav1)\
**Post date:** [November 25, 2020, 10:15am UTC](https://discuss.elastic.co/t/grok-increase-cpu-usage-to-700/256633/1 "2020-11-25T10:15:17Z")

</div>

Hello team,

we have implemented a following GROK multi pattern match but when we deploy the pipeline for this component it raises CPU from 100 to 700%.

Is there a way how to improve its performance?  
One way would be to add more logstash nodes (we have 2).

```auto
grok {
            match => {
                "[message]" => [
                    "^%{IP:[http][remoteIp]} %{HOSTNAME:[http][sourceHostname]} %{NOTSPACE} \[%{HTTPDATE:[http][timeStamp]}\] \"%{WORD:[http][method]} %{URIPATH:[http][uri]} %{DATA:[http][version]}\" %{INT:[http][responseCode]} (?:%{INT:[http][responseSize]}|-) \"%{DATA:[http][referer]}\" \"%{DATA:[http][userAgent]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} rtm:%{NOTSPACE}/%{NUMBER:[http][responseTimeMs]} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} hct:\"%{DATA:[http][contentType]}\" %{NOTSPACE} hac:\"%{DATA:[http][acceptHeaders]}\" hxa:\"%{DATA:[http][requestHeaders]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} sve:%{NOTSPACE:[http][tlsVersion]} sci:%{NOTSPACE:[http][cipher]} %{GREEDYDATA}$",
                    "^%{IP:[http][remoteIp]} %{HOSTNAME:[http][sourceHostname]} %{NOTSPACE} \[%{HTTPDATE:[http][timeStamp]}\] \"%{WORD:[http][method]} %{URIPATH:[http][uri]}%{URIPARAM:[http][queryString]} %{DATA:[http][version]}\" %{INT:[http][responseCode]} (?:%{INT:[http][responseSize]}|-) \"%{DATA:[http][referer]}\" \"%{DATA:[http][userAgent]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} rtm:%{NOTSPACE}/%{NUMBER:[http][responseTimeMs]} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} hct:\"%{DATA:[http][contentType]}\" %{NOTSPACE} hac:\"%{DATA:[http][acceptHeaders]}\" hxa:\"%{DATA:[http][requestHeaders]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} sve:%{NOTSPACE:[http][tlsVersion]} sci:%{NOTSPACE:[http][cipher]} %{GREEDYDATA}$",
                    "^%{IP:[http][remoteIp]} %{HOSTNAME:[http][sourceHostname]} %{NOTSPACE} \[%{HTTPDATE:[http][timeStamp]}\] \"%{WORD:[http][method]} %{GREEDYDATA:[http][uri]} %{DATA:[http][version]}\" %{INT:[http][responseCode]} (?:%{INT:[http][responseSize]}|-) \"%{DATA:[http][referer]}\" \"%{DATA:[http][userAgent]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} rtm:%{NOTSPACE}/%{NUMBER:[http][responseTimeMs]} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} hct:\"%{DATA:[http][contentType]}\" %{NOTSPACE} hac:\"%{DATA:[http][acceptHeaders]}\" hxa:\"%{DATA:[http][requestHeaders]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} sve:%{NOTSPACE:[http][tlsVersion]} sci:%{NOTSPACE:[http][cipher]} %{GREEDYDATA}$",
                    "^%{IP:[http][remoteIp]} %{HOSTNAME:[http][sourceHostname]} %{NOTSPACE} \[%{HTTPDATE:[http][timeStamp]}\] \"%{WORD:[http][method]} %{URIPATH:[http][uri]} %{DATA:[http][version]}\" %{INT:[http][responseCode]} (?:%{INT:[http][responseSize]}|-) \"%{DATA:[http][referer]}\" \"%{DATA:[http][userAgent]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} rtm:%{NOTSPACE}/%{NUMBER:[http][responseTimeMs]} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} hct:\"%{DATA:[http][contentType]}\" hco:%{GREEDYDATA} hac:\"%{DATA:[http][acceptHeaders]}\" hxa:\"%{DATA:[http][requestHeaders]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} sve:%{NOTSPACE:[http][tlsVersion]} sci:%{NOTSPACE:[http][cipher]} %{GREEDYDATA}$",
                    "^%{IP:[http][remoteIp]} %{HOSTNAME:[http][sourceHostname]} %{NOTSPACE} \[%{HTTPDATE:[http][timeStamp]}\] \"%{WORD:[http][method]} %{URIPATH:[http][uri]}%{URIPARAM:[http][queryString]} %{DATA:[http][version]}\" %{INT:[http][responseCode]} (?:%{INT:[http][responseSize]}|-) \"%{DATA:[http][referer]}\" \"%{DATA:[http][userAgent]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} rtm:%{NOTSPACE}/%{NUMBER:[http][responseTimeMs]} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} hct:\"%{DATA:[http][contentType]}\" hco:%{GREEDYDATA} hac:\"%{DATA:[http][acceptHeaders]}\" hxa:\"%{DATA:[http][requestHeaders]}\" %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} sve:%{NOTSPACE:[http][tlsVersion]} sci:%{NOTSPACE:[http][cipher]} %{GREEDYDATA}$"
                ]
            }
            remove_field => ["message"]
            tag_on_failure => ["_grokparsefailure", "_gefeparsefailure"]
        }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 25, 2020, 2:37pm UTC](https://discuss.elastic.co/t/grok-increase-cpu-usage-to-700/256633/2 "2020-11-25T14:37:57Z")

</div>

> [@vaclav1](#):
>
> Is there a way how to improve its performance?

Yes.

Regexps can be cheap when they match, but expensive when they fail to match. This is especially true of GREEDYDATA. You can make your regexps cheaper.

Firstly, you are parsing the same prefix in every pattern. You can parse that once so that you avoid the cost of repeatedly parsing it when the rest of the expression fails to match.

```
grok { match => { "[message]" => "^%{IP:[http][remoteIp]} %{HOSTNAME:[http][sourceHostname]} %{NOTSPACE} \[%{HTTPDATE:[http][timeStamp]}\] %{GREEDYDATA:restOfLine}" } }

```

The match [restOfLine] against your list of anchored patterns.

Secondly, you can use dissect instead of grok, which is much less flexible and therefore cheaper.

```
dissect { mapping => { "[message]" => "%{[http][remoteIp]} %{[http][sourceHostname]} %{} [%{http][timeStamp]}] %{restOfLine}" } }

```

Thirdly, you are using an array of patterns to match the request. You can pull out the request by itself and then use a separate grok to match these patterns against it. Either

```
dissect { mapping => { "restOfLine" => '"%{httpRequest}" %{everythingElse}' } }

```

or

```
grok { match => { "restOfLine" => '"(?<httpRequest>[^"]*)" %{GREEDYDATA:everythingElse}" } }

```

You could then use another dissect or grok to take off status, size, referer and user agent (again, replacing DATA with QS or the custom pattern used for httpRequest).

If I am reading it correctly you then have a variable section followed by rtm: and a fixed set of stuff. You could pull out the fixed section using dissect

```
dissect { mapping => { "everythingElse" => "%{beforeRtm} rtm: %{afterRtm}" } }

```

or grok

```
grok { match => { "everythingElse => "%{DATA:beforeRtm} rtm: %GREEDYDATA:afterRtm}" } }

```

then match your list of variable patterns just against the much short [beforeRtm] field.

---

<div class="post-metadata">

**Author:** ![vaclav1](https://avatars.discourse-cdn.com/v4/letter/v/c68b51/32.png) [@vaclav1](https://discuss.elastic.co/u/vaclav1)\
**Post date:** [November 25, 2020, 2:42pm UTC](https://discuss.elastic.co/t/grok-increase-cpu-usage-to-700/256633/3 "2020-11-25T14:42:28Z")

</div>

Many thanks! Just after first split to common part it dropped to 300% :-))

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 23, 2020, 2:42pm UTC](https://discuss.elastic.co/t/grok-increase-cpu-usage-to-700/256633/4 "2020-12-23T14:42:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.