# Filebeat nginx module not working

**URL:** <https://discuss.elastic.co/t/filebeat-nginx-module-not-working/235170>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 1, 2020, 1:06pm UTC](https://discuss.elastic.co/t/filebeat-nginx-module-not-working/235170 "2020-06-01T13:06:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [June 1, 2020, 1:06pm UTC](https://discuss.elastic.co/t/filebeat-nginx-module-not-working/235170/1 "2020-06-01T13:06:36Z")

</div>

Hi Team,

My Log pipeline is as follows

```
Filebeat_7.6.2-----> Logstash_7.6.2-------->Redis------>Elastic_cloud

```

I have enabled filebeat nginx module and have configured the nginx log paths (access and error)

```
filebeat modules enable nginx

```

My logstash configuration is as follows.

```
input {
  redis {
    data_type => "list"
    key => "filebeat"
    host => "pdsdod-elk-rdsjedis.asdsdsd.0001.euw1.cache.amazonaws.com"
    port => 6379
    db => 1
  }
}
filter {
  if [fileset][module] == "nginx" {
    if [fileset][name] == "access" {
      grok {
        match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\""] }
        remove_field => "message"
       }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
       remove_field => "[nginx][access][time]"
      }
      useragent {
        source => "[nginx][access][agent]"
        target => "[nginx][access][user_agent]"
        remove_field => "[nginx][access][agent]"
      }
      geoip {
        source => "[nginx][access][remote_ip]"
        target => "[nginx][access][geoip]"
      }
    }
    else if [fileset][name] == "error" {
      grok {
        match => { "message" => ["%{DATA:[nginx][error][time]} \[%{DATA:[nginx][error][level]}\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}"] }
        remove_field => "message"
      }
      mutate {
        rename => { "@timestamp" => "read_timestamp" }
      }
      date {
        match => ["[nginx][error][time]", "YYYY/MM/dd H:m:s" ]
        remove_field => "[nginx][error][time]"
      }
    }
  }
}
output {
    elasticsearch {
        hosts => ["https://93xxxxxxxxxxxxxxxxxxx263122.us-east-1.aws.found.io:9243/"]
        user => "elastic"
        password => "xxxxxxxxx"
        index => "filebeat-%{+YYYY.MM.dd}"
        manage_template => false
    }
  }

```

So as per this logstash filter which parses Nginx logs we need to have a field fieldnamed `[fileset]` and `[nginx]` which is missing in the logs.

Entire log is clumsed under the message field and dashboards are just blank.

**Please shed some light on how to fix**

Regards  
Karthik.K

---

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [June 3, 2020, 9:16am UTC](https://discuss.elastic.co/t/filebeat-nginx-module-not-working/235170/2 "2020-06-03T09:16:42Z")

</div>

Hi Admins,

Any one please help me.

Regards  
Karthik.K

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [June 3, 2020, 4:29pm UTC](https://discuss.elastic.co/t/filebeat-nginx-module-not-working/235170/3 "2020-06-03T16:29:17Z")

</div>

Could you please share your Filebeat configuration formatted using `</>` and possibly debug logs?

---

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [June 4, 2020, 1:17pm UTC](https://discuss.elastic.co/t/filebeat-nginx-module-not-working/235170/4 "2020-06-04T13:17:20Z")

</div>

Hi Noemi,

Here is the filebeat config.

```
#=========================== Filebeat inputs ==================================
filebeat.inputs:
- type: log
  processors:
  - if:
      equals:
        log.file.path: "/var/log/nginx/access.log"
    then: 
      - add_tags:
          tags: [nginx_access]
  - if:
      equals:
        log.file.path: "/var/log/nginx/error.log"
    then: 
      - add_tags:
          tags: [nginx_error]

  enabled: true
  paths:
    - /var/log/nginx/access.log
    - /var/log/nginx/error.log

#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml	# Glob pattern for configuration loading
  reload.enabled: true # Set to true to enable config reloading
  reload.period: 10s # Period on which files under path should be checked for changes
#==================== Elasticsearch template setting ==========================
setup.template:
  enabled: true
  overwrite: false
  name: "filebeat"
  pattern: "filebeat-*"
  fields: "/etc/filebeat/fields.yml"
setup.template.settings:
  index.number_of_shards: 1
#================================ General =====================================
tags: ["mni-prod"]
#============================== Dashboards =====================================
setup.dashboards.enabled: true
setup.dashboards.index: "filebeat-*"
#============================== Kibana =========================================
setup.ilm.enabled: false
setup.kibana:
  space.id: "xxxxx-com"
  host: "https://5719exxxxxxxxxxx778bd8.us-east-1.aws.found.io:9243"
  username: "elastic"
  password: "xxxxxxxxxxxx"
#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  enabled: false
  index: "filebeat-%{+yyyy-MM-dd}"
  hosts: ["https://93xxxxxxxxxxxxxxxxx2.us-east-1.aws.found.io:9243/"]
  username: "elastic"
  password: "xxxxxxxxxx"
#--------------------------Redis-------------------------------------------------
output.redis:
  enabled: true
  hosts: ["piop-elk-re.0001.euw1.cache.amazonaws.com:6379"]
  key: filebeat
  db: 0
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

```

Here is my logstash config to parse the nginx logs

```
input {
  redis {
    id => "filebeat_in_pipe"
    data_type => "list"
    key => "filebeat"
    host => "pxxxxxxxxxxxxxxxe.amazonaws.com"
    port => 6379
    db => 0
  }
}
filter {
  if "nginx_access" in [tags] {
    grok {
      match => { "message" => "%{IPORHOST:clientip} (?:-|(%{WORD}.%{WORD})) %{USER:ident} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:useragent} %{QS:real_ip}" }
      remove_field => "message"
    }
    mutate {
      add_field => { "read_timestamp" => "%{@timestamp}" }
      remove_tag => [""]
    }
    date {
      match => ["timestamp", "dd/MMM/YYYY:H:m:s Z"]
      remove_field => "[nginx][access][time]"
    }
    useragent {
      source => "useragent"
      target => "[nginx][access][user_agent]"
      remove_field => ["useragent", "[browser][build]", "[browser][major]", "[browser][minor]", "[browser][os_minor]", "[browser][patch]", "[browser][os]"]
    }
    geoip {
      #database => "/usr/share/logstash/enhancers/maxmind/"
      source => "real_ip"
      target => "[nginx][access][geoip]"
    }
  }
  else if "nginx_error" in [tags] {
    grok {
      match => { "message" => ["%{DATA:[nginx][error][time]} \[%{DATA:[nginx][error][level]}\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}"] }
      remove_field => "message"
    }
    mutate {
      rename => { "@timestamp" => "read_timestamp" }
    }
    date {
      match => ["[nginx][error][time]", "YYYY/MM/dd H:m:s" ]
      remove_field => "[nginx][error][time]"
    }
  }
}
output {
  redis {
    id => "filebeat_out_pipe"
    data_type => "list"
    key	=> "filebeat"
    host => "pxxxxxxxxxxxxxxxxxx1.euw1.cache.amazonaws.com"
    port => 6379
    db => 1
  }

```

}

I have another config which writes from redis to elastic

```
input {
  redis {
    id => "filebeat_in_pipe"
    data_type => "list"
    key => "filebeat"
    host => "pxxxxxxxxxxxxxxx1.cache.amazonaws.com"
    port => 6379
    db => 1
  }
}
output {
  elasticsearch {
    hosts => ["https://9xxxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.aws.found.io:9243/"]
    user => "elastic"
    password => "xxxxxxxxxxxxxx"
    index => "filebeat-%{+YYYY.MM.dd}"
    manage_template => false
  }
stdout {}
}
```

---

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [June 4, 2020, 1:17pm UTC](https://discuss.elastic.co/t/filebeat-nginx-module-not-working/235170/5 "2020-06-04T13:17:57Z")

</div>

Now logs are being piped to elastic. But the dashboard is showing nothing. (No errors/No data while accessing)

---

<div class="post-metadata">

**Author:** ![karthiknpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karthiknpy/32/53587_2.png) [@karthiknpy](https://discuss.elastic.co/u/karthiknpy)\
**Post date:** [June 8, 2020, 6:38am UTC](https://discuss.elastic.co/t/filebeat-nginx-module-not-working/235170/6 "2020-06-08T06:38:55Z")

</div>

Admins, any idea what went wrong here ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2020, 6:39am UTC](https://discuss.elastic.co/t/filebeat-nginx-module-not-working/235170/7 "2020-07-06T06:39:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.