# Grok filter pattern for nginx

**URL:** <https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520>\
**Category:** Logstash\
**Created:** [September 6, 2018, 7:47am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520 "2018-09-06T07:47:30Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [September 6, 2018, 7:47am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/1 "2018-09-06T07:47:30Z")

</div>

Hello Team,

I am using ELK6.4.0 and filebeat6.4.0. Currently i am sending only my application logs over elasticsearch using filebeat and parse via logstash. But now we want to send the nginx logs (error and access) over elasticsearch. But i am confused how i can add the grok pattern for nginx in my current filter.

Please find the current logstash configuration:-

```auto
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/pki/tls/ca.crt"]
    ssl_certificate => "/etc/pki/tls/server.crt"
    ssl_key => "/etc/pki/tls/server.key"
    ssl_verify_mode => "peer"
    tls_min_version => "1.2"
  }
}
filter {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}", "\I\,\s\[(?<date-time>[\d\-\w\:\.]+)\s\#(?<pid>\d+)\]\s+(?<loglevel>\w+)\s\-+\s\:\s\[(?<request-id>[\d\w\-]+)\]\s(?<method>[\w\s]+)\s\"(?<path>[\w\/\.]+)\"\s(?<mlp-message>.*)", "\I\,\s\[(?<date-time>[\d\-\w\:\.]+)\s\#(?<pid>[\d]+)\]\s\s(?<loglevel>[\w]+)\s\--\s\:\s\[(?<request-id>[\d\-\w]+)\]\s(?:[cC]urrent\s)?[dD]evice[\s:]+(?<device-id>[\w\s\:]+)", "\I\,\s\[(?<date-time>[\d\-\w\:\.]+)\s\#(?<pid>\d+)\]\s+(?<loglevel>\w+)\s\-+\s\:\s\[(?<request-id>[\d\w\-]+)\]\s(?<mlp-message>.*)", "\w\,\s\[(?<date-time>[\w\-\:\.]+)\s\#(?<pid>\d+)\]\s+(?<loglevel>\w+)\s(?<mlp-message>.*)" ] }
add_field => ["received_at", "%{@timestamp}"] add_field => ["received_from", "%{host}"]
}
}
output {
  elasticsearch {
    hosts => ["xyz:9200"]
    sniffing => true
    manage_template => false
# index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

Can you please help me how i can add grok pattern for nginx.

Note:- Current filter have default syslog pattern and pattern for my application.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 6, 2018, 8:31am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/2 "2018-09-06T08:31:23Z")

</div>

In your Filebeat configuration set fields that indicate the type of log you have. Then add conditionals to your Logstash configuration to choose between different filters.

[https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals)

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [September 6, 2018, 9:06am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/3 "2018-09-06T09:06:39Z")

</div>

Hello Magnus,

Thank you for your response.

> [@magnusbaeck](#):
>
> In your Filebeat configuration set fields that indicate the type of log you have.

Can you please elaborate little bit more?

Currently i have setup type log in my filebeat input. Please refer the below config part:

```auto
#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
# - /var/log/*.log
    - /var/apps/mobilock/shared/log/production.log

```

> [@magnusbaeck](#):
>
> Then add conditionals to your Logstash configuration to choose between different filters.

Can you please give me small example. I am running out of ideas here, because i never used conditional base filter.

Thank you.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 6, 2018, 10:12am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/4 "2018-09-06T10:12:01Z")

</div>

> Can you please elaborate little bit more?

Here's an example that sets a field named `mycustomvar`: [elasticsearch - Generating filebeat custom fields - Stack Overflow](https://stackoverflow.com/a/37097453/414355)

> Can you please give me small example. I am running out of ideas here, because i never used conditional base filter.

The documentation I linked to earlier contains several examples. For example,

```plaintext
if [action] == "login" {
  mutate { remove_field => "secret" }
}

```

shows how to run a mutate filter only if the `action` field has a particular value.

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [September 6, 2018, 10:18am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/5 "2018-09-06T10:18:08Z")

</div>

Hello Magnus,

Thank you for providing useful info.

I have last question, Can we use filebeat nginx module to send the logs over elasticsearch?

Thanks.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 6, 2018, 10:20am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/6 "2018-09-06T10:20:14Z")

</div>

> Can we use filebeat nginx module to send the logs over elasticsearch?

Yes, most likely.

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [September 6, 2018, 11:14am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/7 "2018-09-06T11:14:19Z")

</div>

Hello Magnus,

Thank you for your support.

I have enabled the filebeat nginx module and started getting the logs on kibana dashboard for nginx.

But i am facing another issue i.e i am getting the nginx logs over kibana dashboard but filebeat nginx dashboard is not showing any data. Please refer the below screenshots ![Selection_024](https://us1.discourse-cdn.com/elastic/original/3X/2/7/274df8bb60fd70b419e82587367314b14ab4be96.png)

![Selection_025](https://us1.discourse-cdn.com/elastic/original/3X/6/8/68ba1c0810aa50e1ee97d5ba649e9fb5ed9d737b.png)

I am using filebeat prospector also for our arbitrary application. is that have any impact?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 6, 2018, 11:51am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/8 "2018-09-06T11:51:00Z")

</div>

I suggest you ask questions about Filebeat in the Filebeat category.

---

<div class="post-metadata">

**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\
**Post date:** [September 6, 2018, 12:04pm UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/9 "2018-09-06T12:04:15Z")

</div>

Sure. Thanks 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 4, 2018, 12:04pm UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx/147520/10 "2018-10-04T12:04:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.