# Add grok filter for costume log data in Filebeat's NGINX module

**URL:** <https://discuss.elastic.co/t/add-grok-filter-for-costume-log-data-in-filebeats-nginx-module/161855>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 21, 2018, 1:59pm UTC](https://discuss.elastic.co/t/add-grok-filter-for-costume-log-data-in-filebeats-nginx-module/161855 "2018-12-21T13:59:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheNmaptomyHeartBeat](https://avatars.discourse-cdn.com/v4/letter/t/47e85d/32.png) [@TheNmaptomyHeartBeat](https://discuss.elastic.co/u/TheNmaptomyHeartBeat)\
**Post date:** [December 21, 2018, 1:59pm UTC](https://discuss.elastic.co/t/add-grok-filter-for-costume-log-data-in-filebeats-nginx-module/161855/1 "2018-12-21T13:59:39Z")

</div>

Hi

I'm not sure if this is the best way to go on about this. If there is a better way, please advice.

I've added a new `access.log` entry for NGINX that tracks the following:

```auto
$remote_addr 
$ssl_protocol 
$ssl_cipher 
$request

```

Because it's not the default log for NGINX, the NGINX module if failing to break down the `message` into the different components. This is because the grok filter doesn't recognize this log format.  
Kibana has the following error:

```auto
Provided Grok expressions do not match field value

```

I looked around for a while and came to the conclusion that I need to add a grok pattern for the new log format to `/usr/share/filebeat/module/nginx/access/ingest/default.json`.

The log and grok patterns are:

```auto
11.11.11.11 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 GET / HTTP/1.1

```

```auto
%{IP:clientip} %{DATA:tlsprotocol} %{DATA:cipher} %{WORD:request}\ / %{GREEDYDATA:http.protocol}

```

The content of `default.json`

```auto
{
  "description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
  "processors": [{
    "grok": {
      "field": "message",
      "patterns":[
        "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\""
        ],
      "pattern_definitions": {
        "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
      },
      "ignore_missing": true
    }
  }, {
    "split": {
      "field": "nginx.access.remote_ip_list",
      "separator": "\"?,?\\s+"
    }
  }, {
    "script": {
      "lang": "painless",
      "inline": "boolean isPrivate(def ip) { try { StringTokenizer tok = new StringTokenizer(ip, '.'); int firstByte = Integer.parseInt(tok.nextToken()); int secondByte = Integer.parseInt(tok.nextToken()); if (firstByte == 10) { return true; } if (firstByte == 192 && secondByte == 168) { return true; } if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) { return true; } if (firstByte == 127) { return true; } return false; } catch (Exception e) { return false; } } def found = false; for (def item : ctx.nginx.access.remote_ip_list) { if (!isPrivate(item)) { ctx.nginx.access.remote_ip = item; found = true; break; } } if (!found) { ctx.nginx.access.remote_ip = ctx.nginx.access.remote_ip_list[0]; }"
      }
  }, {
    "remove":{
      "field": "message"
    }
  }, {
    "rename": {
      "field": "@timestamp",
      "target_field": "read_timestamp"
    }
  }, {
    "date": {
      "field": "nginx.access.time",
      "target_field": "@timestamp",
      "formats": ["dd/MMM/YYYY:H:m:s Z"]
    }
  }, {
    "remove": {
      "field": "nginx.access.time"
    }
  }, {
    "user_agent": {
      "field": "nginx.access.agent",
      "target_field": "nginx.access.user_agent"
    }
  }, {
    "remove": {
      "field": "nginx.access.agent"
    }
  }, {
    "geoip": {
      "field": "nginx.access.remote_ip",
      "target_field": "nginx.access.geoip"
    }
  }],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}

```

I tried adding the grok filter to that but then Filebeat stops, I tried a number of ways. But i'm not sure how to add another pattern to that json file without it breaking.

Thanks for any advice.

---

<div class="post-metadata">

**Author:** ![TheNmaptomyHeartBeat](https://avatars.discourse-cdn.com/v4/letter/t/47e85d/32.png) [@TheNmaptomyHeartBeat](https://discuss.elastic.co/u/TheNmaptomyHeartBeat)\
**Post date:** [December 21, 2018, 3:04pm UTC](https://discuss.elastic.co/t/add-grok-filter-for-costume-log-data-in-filebeats-nginx-module/161855/2 "2018-12-21T15:04:08Z")

</div>

I manged to add the new pattern without it breaking.

```auto
{
  "description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
  "processors": [{
    "grok": {
      "field": "message",
      "patterns":[
        "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\""
        ],
      "pattern_definitions": {
        "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
      },
      "ignore_missing": true
    }
  }, {
    "grok": {
      "field": "message",
      "patterns":[
          "%{IP:client.ip} %{DATA:tlsi.protocol} %{DATA:cipher} %{WORD:request} / %{GREEDYDATA:http.protocol}"
        ],
      "ignore_missing": true
    }
  }, {
    "split": {
      "field": "nginx.access.remote_ip_list",
      "separator": "\"?,?\\s+"
    }
  }, {
    "script": {
      "lang": "painless",
      "inline": "boolean isPrivate(def ip) { try { StringTokenizer tok = new StringTokenizer(ip, '.'); int firstByte = Integer.parseInt(tok.nextToken()); int secondByte = Integer.parseInt(tok.nextToken()); if (firstByte == 10) { return true; } if (firstByte == 192 && secondByte == 168) { return true; } if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) { return true; } if (firstByte == 127) { return true; } return false; } catch (Exception e) { return false; } } def found = false; for (def item : ctx.nginx.access.remote_ip_list) { if (!isPrivate(item)) { ctx.nginx.access.remote_ip = item; found = true; break; } } if (!found) { ctx.nginx.access.remote_ip = ctx.nginx.access.remote_ip_list[0]; }"
      }
  }, {
    "remove":{
      "field": "message"
    }
  }, {
    "rename": {
      "field": "@timestamp",
      "target_field": "read_timestamp"
    }
  }, {
    "date": {
      "field": "nginx.access.time",
      "target_field": "@timestamp",
      "formats": ["dd/MMM/YYYY:H:m:s Z"]
    }
  }, {
    "remove": {
      "field": "nginx.access.time"
    }
  }, {
    "user_agent": {
      "field": "nginx.access.agent",
      "target_field": "nginx.access.user_agent"
    }
  }, {
    "remove": {
      "field": "nginx.access.agent"
    }
  }, {
    "geoip": {
      "field": "nginx.access.remote_ip",
      "target_field": "nginx.access.geoip"
    }
  }],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}
                        

```

And I deleted the ingest pipeline from the dev tools in kibana. The new one is loading.  
but kibana is still showing the grok filter is failing.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 27, 2018, 1:21pm UTC](https://discuss.elastic.co/t/add-grok-filter-for-costume-log-data-in-filebeats-nginx-module/161855/3 "2018-12-27T13:21:51Z")

</div>

You can try to debug your grok/pipeline definition using the [Kibana grok debugger](https://www.elastic.co/guide/en/kibana/current/grokdebugger-getting-started.html) or via Console using the [Ingest Node Simulate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html).

The pipeline definition has more than one grok filter. Using the [Ingest Node Simulate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html) in verbose mode you will get back a document with details of input/output/failure for every single processor in the pipeline.

---

<div class="post-metadata">

**Author:** ![TheNmaptomyHeartBeat](https://avatars.discourse-cdn.com/v4/letter/t/47e85d/32.png) [@TheNmaptomyHeartBeat](https://discuss.elastic.co/u/TheNmaptomyHeartBeat)\
**Post date:** [January 16, 2019, 3:25pm UTC](https://discuss.elastic.co/t/add-grok-filter-for-costume-log-data-in-filebeats-nginx-module/161855/4 "2019-01-16T15:25:58Z")

</div>

Hi Steffens,  
Thanks for assisting me.

I have tried using those tools to get it to work. I removed the original content in `default.json` and replaced it with the following as a test.

```auto
{
  "description": "Pipeline for parsing Tailored Nginx access logs used for TLS.",
  "processors": [{
    "grok": {
      "field": "message",
      "patterns":[
        "%{IP:nginx.access.remote_ip} %{DATA:nginx.access.protocol} %{DATA:nginx.access.cipher} %{WORD:nginx.access.request} / HTTP/%{NUMBER:nginx.access.http_version}"
       ],
      "ignore_missing": true
    }
  }]
}

```

This is the pattern for the costume log formats. When I reloaded the pipeline it worked. the filter broke down the message field.

I have seen in other questions people having [more than one pattern](https://discuss.elastic.co/t/nginx-logs-provided-grok-expressions-do-not-match-field-value/128518/7?u=thenmaptomyheartbeat) in a Filebeat module.

I just can't seem to add the costume log format pattern without it coming up with a syntax error or failing completely.

---

<div class="post-metadata">

**Author:** ![TheNmaptomyHeartBeat](https://avatars.discourse-cdn.com/v4/letter/t/47e85d/32.png) [@TheNmaptomyHeartBeat](https://discuss.elastic.co/u/TheNmaptomyHeartBeat)\
**Post date:** [January 17, 2019, 9:32am UTC](https://discuss.elastic.co/t/add-grok-filter-for-costume-log-data-in-filebeats-nginx-module/161855/5 "2019-01-17T09:32:30Z")

</div>

I was finally able to solve this.

The issue turned out to be a syntax error.

All I had to do was add a comma to the end of the first grok expression in the patterns array, and then add my costume NGINX log expression. This didn't work for me before because of syntax error but I thought it was the Filebeat module not accepting my grok expression.

It's only once I started using a JSON validator that it became much easier.  
This is my final `default.json`

```auto
{
        "description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
        "processors": [{
                "grok": {
                        "field": "message",
                        "patterns": [
                                "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}\" %{NUMBER:nginx.access.responss
e_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\"" EVERYTHING AFTER THE COMMA IS COSTUME PATTERN, "%{IP:nginx.access.remote_ip} %{DATA:nginx.access.protocol} %{DATA:nginx.access.cipher} %{WORD:nginx.access.request} / HTTP/%{NUMBER:nginx.access.http_versioo
n}"" 
                        ],
                        "pattern_definitions": {
                                "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
                        },
                        "ignore_missing": true
                }
        }, {
                "split": {
                        "field": "nginx.access.remote_ip_list",
                        "separator": "\"?,?\\s+"
                }
        }, {
                "script": {
                        "lang": "painless",
                        "inline": "boolean isPrivate(def ip) { try { StringTokenizer tok = new StringTokenizer(ip, '.'); int firstByte = Integer.parseInt(tok.nextToken()); int secondByte = Integer.parseInt(tok.nextToken()); if (firstByte == 10) { retuu
rn true; } if (firstByte == 192 && secondByte == 168) { return true; } if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) { return true; } if (firstByte == 127) { return true; } return fall
se; } catch (Exception e) { return false; } } def found = false; for (def item : ctx.nginx.access.remote_ip_list) { if (!isPrivate(item)) { ctx.nginx.access.remote_ip = item; found = true; break; } } if (!found) { ctxx
.nginx.access.remote_ip = ctx.nginx.access.remote_ip_list[0]; }"
                }
        }, {
                "remove": {
                        "field": "message"
                }
        }, {
                "rename": {
                        "field": "@timestamp",
                        "target_field": "read_timestamp"
                }
        }, {
                "date": {
                        "field": "nginx.access.time",
                        "target_field": "@timestamp",
                        "formats": ["dd/MMM/YYYY:H:m:s Z"]
                }
        }, {
                "remove": {
                        "field": "nginx.access.time"
                }
        }, {
                "user_agent": {
                        "field": "nginx.access.agent",
                        "target_field": "nginx.access.user_agent"
                }
        }, {
                "remove": {
                        "field": "nginx.access.agent"
                }
        }, {
                "geoip": {
                        "field": "nginx.access.remote_ip",
                        "target_field": "nginx.access.geoip"
                }
        }],
        "on_failure": [{
                "set": {
                        "field": "error.message",
                        "value": "{{ _ingest.on_failure_message }}"
                }
        }]
}

```

I then removed the old ingest pipeline from kibana, loaded this new one and restarted filebeat. Works perfectly.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 14, 2019, 9:32am UTC](https://discuss.elastic.co/t/add-grok-filter-for-costume-log-data-in-filebeats-nginx-module/161855/6 "2019-02-14T09:32:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.