# Filter not working filbert nginx module

**URL:** <https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 23, 2018, 6:49pm UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040 "2018-05-23T18:49:50Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\
**Post date:** [May 23, 2018, 6:49pm UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040/1 "2018-05-23T18:49:51Z")

</div>

we are trying to setup central logging system by using Elasticsearch, Kibana and FileBeat. so to start with we are trying to visualise the Nginx logs with the help of Kibana.

we installed Elasticsearch, Kibana and FileBeat of 6.2.4 version. and enabled the Nginx module on filebeat.

below is the filebeat.yml:  
filebeat.prospectors:

- type: log  
enabled: true  
paths:
  - /opt/nginx/logs/_.log  
filebeat.config.modules:  
path: ${path.config}/modules.d/_.yml  
reload.enabled: false  
setup.template.settings:  
index.number\_of\_shards: 3  
setup.kibana:  
host: "172.20.8.206:5601"  
output.elasticsearch:  
hosts: ["172.20.8.206:9200"]  
logging.level: debug

when filebeat is pushing nginx logs to ES index, it is storing nginx logs in single field that is 'message'. below is sample document from filebeat index:

@timestamp:May 23rd 2018, 23:54:20.339 message:172.20.0.18 - - [23/May/2018:13:24:19 -0500] "GET /public/img/favicon/favicon64.png HTTP/1.0" 304 0 "[https://www.datashop.mercyhealthprovider.com/schedule](https://www.datashop.mercyhealthprovider.com/schedule)" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36" prospector.type:log beat.name:cmdsmacods02 beat.hostname:cmdsmacods02 beat.version:6.2.4 source:/opt/nginx/logs/nginx-http\_access.log offset:30,837,260 \_id:vcA-jmMBSBfq-X8bqZ1c \_type:doc \_index:filebeat-6.2.4-2018.05.23 \_score: -

and below is the sample nginx logs:

172.20.0.18 - - [23/May/2018:13:48:26 -0500] "PUT /alpha/api/v2/patients/update\_info?empi=P1309289 HTTP/1.0" 200 64 "[https://www.datashop.mercyhealthprovider.com/patient/P1309289/manual-entry?referrer=CareCoordination&provider=1770699423](https://www.datashop.mercyhealthprovider.com/patient/P1309289/manual-entry?referrer=CareCoordination&provider=1770699423)" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36"  
172.20.0.18 - - [23/May/2018:13:48:27 -0500] "PATCH /alpha/api/v2/patients/P14164129/health\_modules/5af5ac8270bc05382f1bfc19/units/10/tasks/22/complete HTTP/1.0" 200 1256 "[https://www.datashop.mercyhealthprovider.com/patient/P14164129/health-modules](https://www.datashop.mercyhealthprovider.com/patient/P14164129/health-modules)" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"  
172.20.0.18 - - [23/May/2018:13:48:27 -0500] "POST /alpha/api/v2/patients/P1531616/timeline HTTP/1.0" 200 383 "[https://www.datashop.mercyhealthprovider.com/patient/P1531616/health-modules](https://www.datashop.mercyhealthprovider.com/patient/P1531616/health-modules)" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"

due to this we are not able to visualise the logs in default dashboard of Nginx which is provided via filebeat.

any suggestion would be appreciated!

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [May 23, 2018, 7:01pm UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040/2 "2018-05-23T19:01:35Z")

</div>

Could you please format your config and share the debug logs of Filebeat?

---

<div class="post-metadata">

**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\
**Post date:** [May 23, 2018, 7:24pm UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040/3 "2018-05-23T19:24:25Z")

</div>

below is the filebeat.yml

```
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /opt/nginx/logs/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "172.20.8.206:5601"
output.elasticsearch:
  hosts: ["172.20.8.206:9200"]
logging.level: debug'
```

---

<div class="post-metadata">

**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\
**Post date:** [May 23, 2018, 7:30pm UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040/4 "2018-05-23T19:30:09Z")

</div>

sorry, I am not able to attach log file to the issue. can't see any option to attach file.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [May 23, 2018, 7:43pm UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040/5 "2018-05-23T19:43:06Z")

</div>

You need to enable the NGINX module in your config. But do not enable log prospector. Your config should look like this:

```auto
filebeat.modules:
- module: nginx
  access:
    enabled: true
    var.paths:
     - /opt/nginx/logs/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "172.20.8.206:5601"
output.elasticsearch:
  hosts: ["172.20.8.206:9200"]
logging.level: debug'

```

---

<div class="post-metadata">

**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\
**Post date:** [May 24, 2018, 5:55am UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040/6 "2018-05-24T05:55:06Z")

</div>

I did the changes in config file. it still inserting the logs as single line. below is the sample:

```
    2018-05-24T00:52:21.488-0500	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-05-24T05:52:21.488Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4",
    "pipeline": "filebeat-6.2.4-nginx-access-default"
  },
  "offset": 46943419,
  "message": "172.20.0.18 - - [24/May/2018:00:52:20 -0500] \"GET /public/images/favicon/favicon.ico HTTP/1.0\" 304 0 \"https://www.datashop.mercyhealthprovider.com/login\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36\"",
  "source": "/opt/nginx/logs/nginx-http_access.log",
  "fileset": {
    "module": "nginx",
    "name": "access"
  },
  "prospector": {
    "type": "log"
  },
  "beat": {
    "name": "cmdsmacods02",
    "hostname": "cmdsmacods02",
    "version": "6.2.4"
  }
}

```

also I need to index access and error logs both. do I need to define config for access and error logs separately.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [May 24, 2018, 9:13am UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040/7 "2018-05-24T09:13:49Z")

</div>

What is the version of NGINX? I tested your log lines with our current pipeline and I got this message:

```auto
Provided Grok expressions do not match field value

```

Can you see this in your logs?

---

<div class="post-metadata">

**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\
**Post date:** [May 25, 2018, 9:09am UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040/8 "2018-05-25T09:09:18Z")

</div>

we are using nginx version: nginx/1.9.7. and about the logs I am not able to see such logs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2018, 9:20am UTC](https://discuss.elastic.co/t/filter-not-working-filbert-nginx-module/133040/9 "2018-06-22T09:20:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.