Nginx Module

Elastic Stack Beats
1

Hello All,

I have configured metricbeat and enabled the system and Nginx module. The system module is working fine and able to see the metrics on metricbeat dashboard. But no data is available for Nginx.

Config file of my metricbeat:

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#==========================  Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#======================Metric Beat Module=============================
metricbeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    - cpu             # CPU usage
#    - filesystem      # File system usage for each mountpoint
#    - fsstat          # File system summary metrics
    - load            # CPU load averages
    - memory          # Memory usage
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - uptime          # System Uptime
    - core           # Per CPU core usage
    - diskio         # Disk IO
#    - socket         # Sockets and connection info (linux only)
  enabled: true
  period: 20s
  processes: ['.*']
# Configure the metric types that are included by these metricsets.
  cpu.metrics:  ["percentages"]  # The other available options are normalized_percentages and ticks.
  core.metrics: ["percentages"]  # The other available option is ticks.

#-------------------------------- Nginx Module -------------------------------
- module: nginx
  metricsets: ["stubstatus"]
  enabled: true
  period: 10s

  # Nginx hosts
  hosts: ["http://127.0.0.1"]

  # Path to server status. Default server-status
  server_status_path: "server-status"


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "xx.xx.xx.xx:5601"

#============================= Elastic Cloud ==================================

# These settings simplify using metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: []

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["XYZ:7122"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate_authorities: ["/etc/pki/tls/ca.crt"]
  ssl.certificate: "/etc/pki/tls/client.crt"
  ssl.key: "/etc/pki/tls/client.key"
  ssl.key_passphrase: "ky9D=h=w2z2uUCjRqqWF"

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:

Did not get any error in metricbeat logs.

Please help.

Thanks in advance.

2

Could you disable the system module, only enable nginx and then enable debug log level and share it here? This should provide us with much more details.

The same question applies like in the other post: LS config and how did you load the template?

3

Hello Ruflin,

I have disable the system module from config file as well as from modules.d directory. Now only Nginx module is enabled and i have set the loglevel DEBUG. Before disable system moduleI i was getting the metrics properly for same but didn't get the same for nginx.

Please find the logs from metricbeat for nginx:

    "hostname": "xyz"
  },
  "metricset": {
    "rtt": 1949,
    "name": "stubstatus",
    "module": "nginx",
    "host": "127.0.0.1"
  }
}
2018-08-24T04:29:37.339Z	DEBUG	[logstash]	logstash/async.go:142	1 events out of 1 events sent to logstash host xyz:5044. Continue sending
2018-08-24T04:29:46.337Z	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-24T04:29:46.335Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "metricset": {
    "rtt": 1555,
    "name": "stubstatus",
    "module": "nginx",
    "host": "127.0.0.1"
  },
  "error": {
    "message": "HTTP error 403 in stubstatus: 403 Forbidden"
  },
  "beat": {
    "hostname": "orion4",
    "version": "6.2.4",
    "name": "orion4"
  }
}
2018-08-24T04:29:47.338Z	DEBUG	[logstash]	logstash/async.go:142	1 events out of 1 events sent to logstash host xyz:5044. Continue sending
2018-08-24T04:29:56.335Z	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":28},"total":{"ticks":210,"time":224,"value":210},"user":{"ticks":190,"time":196}},"info":{"ephemeral_id":"f282a543-6662-4eb3-8099-1368bdb6674f","uptime":{"ms":270042}},"memstats":{"gc_next":7214416,"memory_alloc":5133104,"memory_total":47458160}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":3,"batches":3,"total":3},"read":{"bytes":105},"write":{"bytes":841}},"pipeline":{"clients":2,"events":{"active":0,"published":3,"total":3},"queue":{"acked":3}}},"metricbeat":{"nginx":{"stubstatus":{"events":3,"failures":3}}},"system":{"load":{"1":0.23,"15":0.5,"5":0.41,"norm":{"1":0.115,"15":0.25,"5":0.205}}}}}}
2018-08-24T04:29:56.336Z	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-08-24T04:29:56.335Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "metricset": {
    "name": "stubstatus",
    "module": "nginx",
    "host": "127.0.0.1",
    "rtt": 1572
  },
  "error": {
    "message": "HTTP error 403 in stubstatus: 403 Forbidden"
  },
  "beat": {
    "name": "xyz",
    "hostname": "xyz",
    "version": "6.2.4"
  }
}

Please find the metricbeat.yml file config

# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#==========================  Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
#  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#======================Metric Beat Module=============================
#metricbeat.modules:

#------------------------------- System Module -------------------------------
#- module: system
#  metricsets:
#    - cpu             # CPU usage
#    - filesystem      # File system usage for each mountpoint
#    - fsstat          # File system summary metrics
#    - load            # CPU load averages
#    - memory          # Memory usage
#    - network         # Network IO
#    - process         # Per process metrics
#    - process_summary # Process summary
#    - uptime          # System Uptime
#    - core           # Per CPU core usage
#    - diskio         # Disk IO
#    - socket         # Sockets and connection info (linux only)
#  enabled: true
#  period: 20s
#  processes: ['.*']
# Configure the metric types that are included by these metricsets.
#  cpu.metrics:  ["percentages"]  # The other available options are normalized_percentages and ticks.
#  core.metrics: ["percentages"]  # The other available option is ticks.

#-------------------------------- Nginx Module -------------------------------
#- module: nginx
#  metricsets: ["stubstatus"]
#  enabled: true
#  period: 10s

  # Nginx hosts
#  hosts: ["http://127.0.0.1"]

  # Path to server status. Default server-status
#  server_status_path: "server-status"
#============================== Kibana =====================================
setup.kibana:
  host: "xx.xx.x.x"
#============================= Elastic Cloud ==================================

# These settings simplify using metricbeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: []

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["logstash:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate_authorities: ["/etc/pki/tls/ca.crt"]
  ssl.certificate: "/etc/pki/tls/client.crt"
  ssl.key: "/etc/pki/tls/client.key"
  ssl.key_passphrase: "ky9D=h=w2z2uUCjRqqWF"
#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.selectors: ["*"]

List of enabled modules:

Enabled:
nginx

Disabled:
aerospike
apache
ceph
couchbase
docker
dropwizard
elasticsearch
etcd
golang
graphite
haproxy
http
jolokia
kafka
kibana
kubernetes
logstash
memcached
mongodb
mysql
php_fpm
postgresql

Please help me to fix this issue.

Thanks.

4

As you can see in the logs, nginx returns "message": "HTTP error 403 in stubstatus: 403 Forbidden". I assume this could be that you require username and password to fetch the data from nginx?

5

Hello Ruflin,

Thank you for your response.

Can you please let me know, you are talking about which username and password and where i need to insert them?

Thanks.

6

Hello Ruflin,

This issue has been resolved. Earlier i was using wrong port in nginx.yml module for host. Now i have changed it and its working fine, getting metrics over kibana in nginx dashboard and visualization.

nginx.yml configuration:

- module: nginx
  metricsets: ["stubstatus"]
  period: 10s

  # Nginx hosts
  hosts: ["http://127.0.0.1:8081"]

  # Path to server status. Default server-status
  server_status_path: "nginx_status"

Added the below configuration in server block in nginx configuration for particular site:

location /nginx_status {
          stub_status on;
          access_log   off;
          allow 127.0.0.1;
          deny all;
        }

Thanks.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.