# Varnish grok pattern optimization

**URL:** <https://discuss.elastic.co/t/varnish-grok-pattern-optimization/52100>\
**Category:** Logstash\
**Created:** [June 7, 2016, 3:21pm UTC](https://discuss.elastic.co/t/varnish-grok-pattern-optimization/52100 "2016-06-07T15:21:51Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![michbsd](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@michbsd](https://discuss.elastic.co/u/michbsd)\
**Post date:** [June 7, 2016, 3:21pm UTC](https://discuss.elastic.co/t/varnish-grok-pattern-optimization/52100/1 "2016-06-07T15:21:51Z")

</div>

Hi,

I am looking for help in optimizing my grok pattern for my varnishncsa logs.

This is what I have now:

`%{IP:ip1}, %{IP:ip2} - \[%{HTTPDATE:timestamp}\] %{WORD:method} '%{URIHOST:host}' '%{PATH:path}' '(?:%{URIPARAM:param}|)' %{NUMBER:http_status} (?:%{NUMBER:bytes}|-') '(?:%{URI:referrer}|-)' %{QS:agent} %{BASE10NUM:berespms} %{WORD:cache_handling} (?:%{QS:jafapp}|-)"`

Here is an example of the log output:

`91.41.222.50, 172.30.1.1 - [07/Jun/2016:17:11:30 +0200] GET 'www.myhost.com' '/the/path/' '' 200 155725 'http://www.my_referrer.com/another/path/' 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko' 0.000049 hit '-'`

The current pattern makes logstash use all available CPU and eventually hang.

I have about of 70-80 log lines per second, shipped from filebeat (via a redis broker)

Any help would be much appreciated.

thanks

Mike

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 12, 2016, 10:00am UTC](https://discuss.elastic.co/t/varnish-grok-pattern-optimization/52100/2 "2016-06-12T10:00:41Z")

</div>

Logstash hanging because of expensive regexps sounds weird. One thing that _might_ help with the CPU load is replacing `'%{SOMEPATTERN:...}'` with `'(?<...>[^']+)'`. That could save the regexp engine from having to backtrack. It can simply continue until it finds the next single quote.

---

<div class="post-metadata">

**Author:** ![michbsd](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@michbsd](https://discuss.elastic.co/u/michbsd)\
**Post date:** [June 15, 2016, 2:02pm UTC](https://discuss.elastic.co/t/varnish-grok-pattern-optimization/52100/3 "2016-06-15T14:02:01Z")

</div>

Hi Magnus,

Thanks for the reply.  
Actually I wrote a manual parser en ruby, in order to identify the culprit.

It seems that the '(?:%{URI:referrer}|-)' did not like referrers with the following format "android-app://some-android-store-url.com" - and would block, use all available for the CPU.

I changed it to URI to NOTSPACE and the problem went away.

Pretty fragile stuff.

Mike

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:52am UTC](https://discuss.elastic.co/t/varnish-grok-pattern-optimization/52100/4 "2017-07-06T04:52:42Z")

</div>