Metricbeat on NGINX reads only access.log and error.log

Elastic Observability Metrics
1

Hi all,
I'm having a strange behavior, my metricbeat is reading only

  • /var/log/nginx/access.log
  • /var/log/nginx/error.log

I have different .log file per each location on my NGINX.
this is what I configured in my /etc/metricbeat/modules.d/nginx.yml

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.6/metricbeat-module-nginx.html

- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/*access.log*"]
  error:
    enabled: true
    var.paths: ["/path/to/log/nginx/*error.log*"]

  metricsets:
    - stubstatus
  period: 10s

  # Nginx hosts
  hosts: ["http://127.0.0.1"]

  # Path to server status. Default server-status
  server_status_path: "basic_status"

any help on this?

Thanks!

2

Hi @carmine.fabrizio,

Could you post a listing of your /var/log/nginx/ folder?

3 
root@dev-nginx01:[~]# ls /var/log/nginx/
access.log  error.log  es_access.log  es_error.log
root@dev-nginx01:[~]#

Thanks

4

Thanks for the listing!

I re-read your question again and I saw that you mentioned metricbeat. Logs are ingested into the elastic stack with filebeat, so maybe the issue is there.

Could you check for me if filebeat is configured, and what is the nginx module configuration?

5

filebeat does not read neither

Mar 12 07:21:39 dev-nginx01.isoftbet.com filebeat[19051]: 2020-03-12T07:21:39.866-0400        INFO        log/input.go:152        Configured paths: [/var/log/nginx/access.log*]
Mar 12 07:21:39 dev-nginx01.isoftbet.com filebeat[19051]: 2020-03-12T07:21:39.867-0400        INFO        log/input.go:152        Configured paths: [/var/log/nginx/error.log*]
Mar 12 07:21:39 dev-nginx01.isoftbet.com filebeat[19051]: 2020-03-12T07:21:39.874-0400        INFO        log/input.go:152        Configured paths: [/var/log/messages* /var/log/syslog*]
Mar 12 07:21:39 dev-nginx01.isoftbet.com filebeat[19051]: 2020-03-12T07:21:39.876-0400        INFO        log/input.go:152        Configured paths: [/var/log/auth.log* /var/log/secure*]
Mar 12 07:21:39 dev-nginx01.isoftbet.com filebeat[19051]: 2020-03-12T07:21:39.876-0400        INFO        crawler/crawler.go:106        Loading and starting Inputs completed. Enabled inputs: 0
Mar 12 07:21:39 dev-nginx01.isoftbet.com filebeat[19051]: 2020-03-12T07:21:39.876-0400        INFO        cfgfile/reload.go:171        Config reloader started
Mar 12 07:21:39 dev-nginx01.isoftbet.com filebeat[19051]: 2020-03-12T07:21:39.880-0400        INFO        log/input.go:152        Configured paths: [/var/log/nginx/access.log*]
Mar 12 07:21:39 dev-nginx01.isoftbet.com filebeat[19051]: 2020-03-12T07:21:39.880-0400        INFO        log/input.go:152        Configured paths: [/var/log/nginx/error.log*]

this is my filebeat.yml

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    - /var/log/nginx/*

also

root@dev-nginx01:[~]# filebeat modules list
Enabled:
nginx
system
6

I think we are getting close. The first two lines on your output mention:

Mar 12 07:21:39 [...]        Configured paths: [/var/log/nginx/access.log*]
Mar 12 07:21:39 [...]        Configured paths: [/var/log/nginx/error.log*]

The patterns there should be /var/log/nginx/*access.log* and /var/log/nginx/*error.log* so the es_ prefixed files in the listing that you shared with me will be included.

Next to your filebeat.yml there should be a modules.d/nginx.yml. If they are not set already, try adding the necessary var.paths configuration there.

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.3/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true
    var.paths: ["/var/log/nginx/*access.log*/"]

  # Error logs
  error:
    enabled: true
    var.paths: ["/var/log/nginx/*error.log*"]

Let me know

7

Yup, it works thanks a lot!

1 Like