Filter not working filbert nginx module

Elastic Stack Beats
1

we are trying to setup central logging system by using Elasticsearch, Kibana and FileBeat. so to start with we are trying to visualise the Nginx logs with the help of Kibana.

we installed Elasticsearch, Kibana and FileBeat of 6.2.4 version. and enabled the Nginx module on filebeat.

below is the filebeat.yml:
filebeat.prospectors:

  • type: log
    enabled: true
    paths:
    • /opt/nginx/logs/.log
      filebeat.config.modules:
      path: ${path.config}/modules.d/      .yml
      reload.enabled: false
      setup.template.settings:
      index.number_of_shards: 3
      setup.kibana:
      host: "172.20.8.206:5601"
      output.elasticsearch:
      hosts: ["172.20.8.206:9200"]
      logging.level: debug

when filebeat is pushing nginx logs to ES index, it is storing nginx logs in single field that is 'message'. below is sample document from filebeat index:

@timestamp:May 23rd 2018, 23:54:20.339 message:172.20.0.18 - - [23/May/2018:13:24:19 -0500] "GET /public/img/favicon/favicon64.png HTTP/1.0" 304 0 "https://www.datashop.mercyhealthprovider.com/schedule" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36" prospector.type:log beat.name:cmdsmacods02 beat.hostname:cmdsmacods02 beat.version:6.2.4 source:/opt/nginx/logs/nginx-http_access.log offset:30,837,260 _id:vcA-jmMBSBfq-X8bqZ1c _type:doc _index:filebeat-6.2.4-2018.05.23 _score: -

and below is the sample nginx logs:

172.20.0.18 - - [23/May/2018:13:48:26 -0500] "PUT /alpha/api/v2/patients/update_info?empi=P1309289 HTTP/1.0" 200 64 "https://www.datashop.mercyhealthprovider.com/patient/P1309289/manual-entry?referrer=CareCoordination&provider=1770699423" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36"
172.20.0.18 - - [23/May/2018:13:48:27 -0500] "PATCH /alpha/api/v2/patients/P14164129/health_modules/5af5ac8270bc05382f1bfc19/units/10/tasks/22/complete HTTP/1.0" 200 1256 "https://www.datashop.mercyhealthprovider.com/patient/P14164129/health-modules" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
172.20.0.18 - - [23/May/2018:13:48:27 -0500] "POST /alpha/api/v2/patients/P1531616/timeline HTTP/1.0" 200 383 "https://www.datashop.mercyhealthprovider.com/patient/P1531616/health-modules" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"

due to this we are not able to visualise the logs in default dashboard of Nginx which is provided via filebeat.

any suggestion would be appreciated!

2

Could you please format your config and share the debug logs of Filebeat?

3

below is the filebeat.yml

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /opt/nginx/logs/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "172.20.8.206:5601"
output.elasticsearch:
  hosts: ["172.20.8.206:9200"]
logging.level: debug'
4

sorry, I am not able to attach log file to the issue. can't see any option to attach file.

5

You need to enable the NGINX module in your config. But do not enable log prospector. Your config should look like this:

filebeat.modules:
- module: nginx
  access:
    enabled: true
    var.paths:
     - /opt/nginx/logs/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "172.20.8.206:5601"
output.elasticsearch:
  hosts: ["172.20.8.206:9200"]
logging.level: debug'
6

I did the changes in config file. it still inserting the logs as single line. below is the sample:

    2018-05-24T00:52:21.488-0500	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-05-24T05:52:21.488Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4",
    "pipeline": "filebeat-6.2.4-nginx-access-default"
  },
  "offset": 46943419,
  "message": "172.20.0.18 - - [24/May/2018:00:52:20 -0500] \"GET /public/images/favicon/favicon.ico HTTP/1.0\" 304 0 \"https://www.datashop.mercyhealthprovider.com/login\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36\"",
  "source": "/opt/nginx/logs/nginx-http_access.log",
  "fileset": {
    "module": "nginx",
    "name": "access"
  },
  "prospector": {
    "type": "log"
  },
  "beat": {
    "name": "cmdsmacods02",
    "hostname": "cmdsmacods02",
    "version": "6.2.4"
  }
}

also I need to index access and error logs both. do I need to define config for access and error logs separately.

7

What is the version of NGINX? I tested your log lines with our current pipeline and I got this message:

Provided Grok expressions do not match field value

Can you see this in your logs?

8

we are using nginx version: nginx/1.9.7. and about the logs I am not able to see such logs.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.