# Optimizing Grok patterns

**URL:** <https://discuss.elastic.co/t/optimizing-grok-patterns/123256>\
**Category:** Logstash\
**Created:** [March 9, 2018, 11:45am UTC](https://discuss.elastic.co/t/optimizing-grok-patterns/123256 "2018-03-09T11:45:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![burnersk](https://avatars.discourse-cdn.com/v4/letter/b/e79b87/32.png) [@burnersk](https://discuss.elastic.co/u/burnersk)\
**Post date:** [March 9, 2018, 11:45am UTC](https://discuss.elastic.co/t/optimizing-grok-patterns/123256/1 "2018-03-09T11:45:47Z")

</div>

I have the following Grok pattern for parsing nginx's error log file:

```
input {
  file {
    id => "nginx_error_file_input"
    codec => "plain"
    tags => ["nginx", "nginx_errors"]
    type => "nginx_error_log"
    path => ["/var/log/nginx/error.log"]
  }
}

filter {
  if [type] == "nginx_error_log" {
    grok {
      id => "nginx_error_grok_filter"
      patterns_dir => ["/etc/logstash/patterns"]
      match => { message => "^%{NGINXTIMESTAMP:timestamp}%{SPACE}\[%{WORD:error_level}\]%{SPACE}%{POSINT:pid}#%{POSINT:tid}:%{SPACE}(\*(?<cid>\d+))?%{SPACE}%{GREEDYDATA:error_message}$" }
      add_tag => ["_nginx_error_grok_filter"]
      tag_on_failure => ["_grokparsefailure_nginx_error_grok_filter"]
    }
    date {
      id => "nginx_error_date_filter"
      match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
      locale => "en-US"
      timezone => "Europe/Berlin"
      target => "@timestamp"
    }
  }
  if "_nginx_error_grok_filter" in [tags] {
    if [error_message] =~ /,\s+client:\s+/ {
      grok {
        id => "nginx_error_client_grok_filter"
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "error_message" => ",%{SPACE}client:%{SPACE}%{IPORHOST:client}" }
        tag_on_failure => ["_grokparsefailure_nginx_error_client_grok_filter"]
      }
    }
    if [error_message] =~ /,\s+server:\s+/ {
      grok {
        id => "nginx_error_server_grok_filter"
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "error_message" => ",%{SPACE}server:%{SPACE}%{IPORHOST:server}" }
        add_tag => ["server_%{server}"]
        tag_on_failure => ["_grokparsefailure_nginx_error_server_grok_filter"]
      }
    }
    if [error_message] =~ /,\s+host:\s+/ {
      grok {
        id => "nginx_error_host_grok_filter"
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "error_message" => ",%{SPACE}host:%{SPACE}%{IPORHOST:host}" }
        add_tag => ["host_%{host}"]
        tag_on_failure => ["_grokparsefailure_nginx_error_host_grok_filter"]
      }
    }
    if [error_message] =~ /,\s+request:\s+/ {
      grok {
        id => "nginx_error_request_grok_filter"
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "error_message" => ',%{SPACE}request:%{SPACE}"%{QS:request}"' }
        tag_on_failure => ["_grokparsefailure_nginx_error_request_grok_filter"]
      }
    }
    if [error_message] =~ /,\s+upstream:\s+/ {
      grok {
        id => "nginx_error_upstream_grok_filter"
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "error_message" => ',%{SPACE}upstream:%{SPACE}"%{QS:upstream}"' }
        tag_on_failure => ["_grokparsefailure_nginx_error_upstream_grok_filter"]
      }
    }
    if [error_message] =~ /,\s+referrer:\s+/ {
      grok {
        id => "nginx_error_referrer_grok_filter"
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "error_message" => ',%{SPACE}referrer:%{SPACE}"%{QS:referrer}"' }
        tag_on_failure => ["_grokparsefailure_nginx_error_referrer_grok_filter"]
      }
    }
  }
}

```

As you can see, the configuration heavily relies on conditional Grok filters, which I guess have performance penalties (assumption). And the readability is not that nice.

Is there a way to optimize this Grok pattern to be more readable and (possibly) more performant.

---

<div class="post-metadata">

**Author:** ![vinodr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinodr/32/28759_2.png) [@vinodr](https://discuss.elastic.co/u/vinodr)\
**Post date:** [March 14, 2018, 3:01am UTC](https://discuss.elastic.co/t/optimizing-grok-patterns/123256/2 "2018-03-14T03:01:37Z")

</div>

@burnersk Can you try using just the first grok filter with or (|) conditions in the match pattern for the various error messages? Something similar to below:

```
  filter {
    if [type] == "nginx_error_log" {
      grok {
      id => "nginx_error_grok_filter"
      patterns_dir => ["/etc/logstash/patterns"]
      match => { message => "^%{NGINXTIMESTAMP:timestamp}%{SPACE}[%{WORD:error_level}]%{SPACE}%{POSINT:pid}#%{POSINT:tid}:%{SPACE}(*(?\d+))?%{SPACE}((,%{SPACE}client:%{SPACE}%{IPORHOST:client})|(,%{SPACE}server:%{SPACE}%{IPORHOST:server})|(,%{SPACE}host:%{SPACE}%{IPORHOST:host})|(,%{SPACE}request:%{SPACE}"%{QS:request})|(,%{SPACE}upstream:%{SPACE}"%{QS:upstream})|(,%{SPACE}referrer:%{SPACE}"%{QS:referrer}))$" }
      }
      date {
        id => "nginx_error_date_filter"
        match => ["timestamp", "yyyy/MM/dd HH:mm:ss"]
        locale => "en-US"
        timezone => "Europe/Berlin"
        target => "@timestamp"
      }
    }
  }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 11, 2018, 3:01am UTC](https://discuss.elastic.co/t/optimizing-grok-patterns/123256/3 "2018-04-11T03:01:48Z")

</div>

Closed on April 11, 2018, 3:01am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.