Hello all, I'm trying to use Metricbeat to monitor my elasticstack. Logstash, Elasticsearch, Kibana, and some Beats are all running inside of docker containers. I am wondering what is the best way to use Metricbeat to monitor these processes?
I believe I have mounted the filesystems correctly and am running Metricbeat with the correct options, however I am unsure how to combine the Hints Based Autodiscover with the Modules and Dynamic Config Reloading.
Do I need to define my modules in the modules.d directory? Do I need to define them as Templates under Hint Based Autodiscover? Will it work if I simply define them as Modules in the metricbeat.yml configuration file? Will these work with the Dynamic Config Reloading?
Here is my configuration for metricbeat.yml
########################## Metricbeat Configuration ###########################
# This file is a full configuration example documenting all non-deprecated
# options in comments. For a shorter configuration example, that contains only
# the most common options, please see metricbeat.yml in the same directory.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html
#============================ Config Reloading ===============================
# Config reloading allows to dynamically load modules. Each file which is
# monitored must contain one or multiple modules as a list.
metricbeat.config.modules:
# Glob pattern for configuration reloading
path: ${path.config}/modules.d/*.yml
# Period on which files under path should be checked for changes
reload.period: 10s
# Set to true to enable config reloading
reload.enabled: true
# Maximum amount of time to randomly delay the start of a metricset. Use 0 to
# disable startup delay.
metricbeat.max_start_delay: 10s
#============================== Autodiscover ===================================
# Autodiscover allows you to detect changes in the system and spawn new modules
# as they happen.
metricbeat.autodiscover:
# List of enabled autodiscover providers
providers:
- type: docker
hints.enabled: true
#========================== Modules configuration =============================
metricbeat.modules:
#-------------------------------- System Module --------------------------------
- module: system
metricsets:
- cpu # CPU usage
- load # CPU load averages
- memory # Memory usage
- network # Network IO
- process # Per process metrics
- process_summary # Process summary
- uptime # System Uptime
- socket_summary # Socket summary
#- core # Per CPU core usage
#- diskio # Disk IO
#- filesystem # File system usage for each mountpoint
#- fsstat # File system summary metrics
#- raid # Raid
#- socket # Sockets and connection info (linux only)
- service # systemd service information
enabled: true
period: 10s
processes: ['.*']
# Configure the metric types that are included by these metricsets.
cpu.metrics: ["percentages","normalized_percentages"] # The other available option is ticks.
core.metrics: ["percentages"] # The other available option is ticks.
# A list of filesystem types to ignore.
#filesystem.ignore_types: []
process.cgroups.enabled: true
#--------------------------------- Beat Module ---------------------------------
- module: beat
metricsets:
- stats
- state
period: 10s
hosts: ["http://localhost:5066"]
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
#ssl.certificate: "/etc/pki/client/cert.pem"
#ssl.key: "/etc/pki/client/cert.key"
# Set to true to send data collected by module to X-Pack
# Monitoring instead of metricbeat-* indices.
#xpack.enabled: false
#-------------------------------- Docker Module --------------------------------
- module: docker
metricsets:
- "container"
- "cpu"
- "diskio"
- "event"
- "healthcheck"
- "info"
#- "image"
- "memory"
- "network"
hosts: ["unix:///var/run/docker.sock"]
period: 10s
enabled: true
# If set to true, replace dots in labels with `_`.
#labels.dedot: false
# If set to true, collects metrics per core.
#cpu.cores: true
# To connect to Docker over TLS you must specify a client and CA certificate.
#ssl:
# certificate_authority: "/etc/pki/root/ca.pem"
# certificate: "/etc/pki/client/cert.pem"
# key: "/etc/pki/client/cert.key"
#---------------------------- Elasticsearch Module ----------------------------
- module: elasticsearch
metricsets:
- node
- node_stats
#- index
#- index_recovery
#- index_summary
#- shard
#- ml_job
period: 10s
hosts: ["${ELASTICSEARCH_HOST}"]
username: ${ES_USER}
password: ${ES_PASSWORD}
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Set to false to fetch all entries
#index_recovery.active_only: true
# Set to true to send data collected by module to X-Pack
# Monitoring instead of metricbeat-* indices.
xpack.enabled: false
#-------------------------------- Kibana Module --------------------------------
- module: kibana
metricsets: ["status"]
period: 10s
hosts: ["${KIBANA_HOST}"]
username: "${ES_USER}"
password: "${ES_PASSWORD}"
basepath: ""
enabled: true
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
# Set to true to send data collected by module to X-Pack
# Monitoring instead of metricbeat-* indices.
xpack.enabled: false
#-------------------------------- Linux Module --------------------------------
- module: linux
period: 10s
metricsets:
- "pageinfo"
# - ksm
# - conntrack
enabled: true
hostfs: /hostfs
#------------------------------- Logstash Module -------------------------------
- module: logstash
metricsets: ["node", "node_stats"]
enabled: true
period: 10s
hosts: ["${LOGSTASH_HOST2}"]
# ================================== General ===================================
# The following example enriches each event with docker metadata, it matches
# container id from log path available in `source` field (by default it expects
# it to be /var/lib/docker/containers/*/*.log).
processors:
- add_docker_metadata: ~
# The following example enriches each event with host metadata.
processors:
- add_host_metadata: ~
# The following example enriches each event with process metadata using
# process IDs included in the event.
processors:
- add_process_metadata:
match_pids: ["system.process.ppid"]
target: system.process.parent
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
enabled: false
hosts: ["${ELASTICSEARCH_HOST}"]
username: ${ES_USER}
password: ${ES_PASSWORD}
ssl.enabled: true
ssl.verification_mode: full
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# Boolean flag to enable or disable the output module.
enabled: true
hosts: ["${LOGSTASH_HOST}"]
ssl.enabled: true
ssl.verification_mode: full
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "${KIBANA_HOST}"
username: ${ES_USER}
password: ${ES_PASSWORD}
ssl.enabled: true
ssl.verification_mode: full
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key
When I check Kibana Metrics, I can see only 1 host (Metricbeat). When I check the docker containers, all of my containers do appear but it appears like it isn't applying any modules to them.
metricbeat_1 | 2020-09-01T16:34:04.396Z INFO module/wrapper.go:259 Error fetching data for metricset beat.state: failure to apply state schema: 1 error: key `module` not found
I'm not sure what I'm doing wrong in my configurations, and help is greatly appreciated!
You can use Hints based autodiscover to monitor the services that run as containers (kibana, logstash etc). Note that you need to annotate these containers with the proper labels so as to make autodiscover able to identify how you want to monitor them (with the respective Metricbeat module). In this please have a look at the respective docs: https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover-hints.html#_docker_3
With autodiscover you should be able to launch the proper modules to monitor the services that run as containers. Let me know if you have any questions about these!
I know that that label is for the "hint based" setting but I hope that this is a good hybrid so I can identify which services are in which container even if I do a custom image for them.
My new metricbeat config is:
########################## Metricbeat Configuration ###########################
# This file is a full configuration example documenting all non-deprecated
# options in comments. For a shorter configuration example, that contains only
# the most common options, please see metricbeat.yml in the same directory.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html
#============================ Config Reloading ===============================
# Config reloading allows to dynamically load modules. Each file which is
# monitored must contain one or multiple modules as a list.
metricbeat.config.modules:
# Glob pattern for configuration reloading
path: ${path.config}/modules.d/*.yml
# Period on which files under path should be checked for changes
reload.period: 10s
# Set to true to enable config reloading
reload.enabled: false
# Maximum amount of time to randomly delay the start of a metricset. Use 0 to
# disable startup delay.
metricbeat.max_start_delay: 10s
#============================== Autodiscover ===================================
# Autodiscover allows you to detect changes in the system and spawn new modules
# as they happen.
metricbeat.autodiscover:
# List of enabled autodiscover providers
providers:
- type: docker
hints.enabled: false
templates:
#---------------------------- Elasticsearch Module ----------------------------
- condition:
contains:
docker.container.labels.co.elastic.metrics/module: "elasticsearch"
config:
- module: elasticsearch
metricsets:
- node
- node_stats
#- index
#- index_recovery
#- index_summary
#- shard
#- ml_job
period: 10s
hosts: ["${ELASTICSEARCH_HOST}"]
username: ${ES_USER}
password: ${ES_PASSWORD}
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
xpack.enabled: false
#--------------------------------- Beat Module ---------------------------------
- condition:
contains:
docker.container.labels.co.elastic.metrics/module: "beat"
config:
- module: beat
metricsets:
- stats
- state
period: 10s
hosts: ["http://localhost:5066"]
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
#ssl.certificate: "/etc/pki/client/cert.pem"
#ssl.key: "/etc/pki/client/cert.key"
# Set to true to send data collected by module to X-Pack
# Monitoring instead of metricbeat-* indices.
#xpack.enabled: false
#-------------------------------- Kibana Module --------------------------------
- condition:
contains:
docker.container.labels.co.elastic.metrics/module: "kibana"
config:
- module: kibana
metricsets: ["status"]
period: 10s
hosts: ["${KIBANA_HOST}"]
username: "${ES_USER}"
password: "${ES_PASSWORD}"
basepath: ""
enabled: true
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
# Set to true to send data collected by module to X-Pack
# Monitoring instead of metricbeat-* indices.
xpack.enabled: false
#------------------------------- Logstash Module -------------------------------
- condition:
contains:
docker.container.labels.co.elastic.metrics/module: "logstash"
config:
- module: logstash
metricsets: ["node", "node_stats"]
enabled: true
period: 10s
hosts: ["${LOGSTASH_HOST2}"]
#========================== Modules configuration =============================
metricbeat.modules:
#-------------------------------- System Module --------------------------------
- module: system
metricsets:
- cpu # CPU usage
- load # CPU load averages
- memory # Memory usage
- network # Network IO
- process # Per process metrics
- process_summary # Process summary
- uptime # System Uptime
- socket_summary # Socket summary
#- core # Per CPU core usage
#- diskio # Disk IO
#- filesystem # File system usage for each mountpoint
#- fsstat # File system summary metrics
#- raid # Raid
#- socket # Sockets and connection info (linux only)
- service # systemd service information
enabled: true
period: 10s
processes: ['.*']
# Configure the metric types that are included by these metricsets.
cpu.metrics: ["percentages","normalized_percentages"] # The other available option is ticks.
core.metrics: ["percentages"] # The other available option is ticks.
# A list of filesystem types to ignore.
#filesystem.ignore_types: []
process.cgroups.enabled: true
# Configure reverse DNS lookup on remote IP addresses in the socket metricset.
#socket.reverse_lookup.enabled: false
#socket.reverse_lookup.success_ttl: 60s
#socket.reverse_lookup.failure_ttl: 60s
# Filter systemd services by status or sub-status
#service.state_filter: ["active"]
# Filter systemd services based on a name pattern
#service.pattern_filter: ["ssh*", "nfs*"]
#-------------------------------- Docker Module --------------------------------
- module: docker
metricsets:
- "container"
- "cpu"
- "diskio"
- "event"
- "healthcheck"
- "info"
#- "image"
- "memory"
- "network"
hosts: ["unix:///var/run/docker.sock"]
period: 10s
enabled: true
# If set to true, replace dots in labels with `_`.
#labels.dedot: false
# If set to true, collects metrics per core.
#cpu.cores: true
# To connect to Docker over TLS you must specify a client and CA certificate.
#ssl:
# certificate_authority: "/etc/pki/root/ca.pem"
# certificate: "/etc/pki/client/cert.pem"
# key: "/etc/pki/client/cert.key"
#-------------------------------- Linux Module --------------------------------
- module: linux
period: 10s
metricsets:
- "pageinfo"
# - ksm
# - conntrack
enabled: true
hostfs: /hostfs
# ================================== General ===================================
# The following example enriches each event with docker metadata, it matches
# container id from log path available in `source` field (by default it expects
# it to be /var/lib/docker/containers/*/*.log).
processors:
- add_docker_metadata: ~
# The following example enriches each event with host metadata.
processors:
- add_host_metadata: ~
# The following example enriches each event with process metadata using
# process IDs included in the event.
processors:
- add_process_metadata:
match_pids: ["system.process.ppid"]
target: system.process.parent
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
enabled: false
hosts: ["${ELASTICSEARCH_HOST}"]
username: ${ES_USER}
password: ${ES_PASSWORD}
ssl.enabled: true
ssl.verification_mode: full
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# Boolean flag to enable or disable the output module.
enabled: true
hosts: ["${LOGSTASH_HOST}"]
ssl.enabled: true
ssl.verification_mode: full
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "${KIBANA_HOST}"
username: ${ES_USER}
password: ${ES_PASSWORD}
ssl.enabled: true
ssl.verification_mode: full
ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
ssl.certificate: "/etc/pki/client/cert.pem"
ssl.key: "/etc/pki/client/cert.key"
Does this look correct to you? I'm not sure which modules are still needed to be defined outside of it or if the linux/docker modules are needed, or if I need to define the modules for the things I want autodiscovered outside of the "template" part of the autodiscover. I am still getting this strange error in my logs: INFO module/wrapper.go:259 Error fetching data for metricset beat.state: failure to apply state schema: 1 error: key 'module' not found, in addition still only the "Metricbeat System" host shows up when I look under the Kibana Metricbeat System ECS dashboard. Same for "Containers Overview" dashboard, it appears that "Metricbeat System" is the only one that appears. I'm not sure if this is normal or showing there's something misconfigured. Under "Metrics", Metricbeat shows up as the only host but the other containers are visible, but they don't show up the same way that Elasticsearch would if I was monitoring it outside of a container.
Thank you for the help!
You should not mix hints/annotations with static templates. If you want to annotate the containers and make metricbeat identify them and start monitoring them with the proper modules you need to enable hints-based autodiscover:
hints.enabled: true
Then Metricbeat will be able to catch the conteiners' labels and start monitoring them based on their labels. So templates: should be removed.
Otherwise you can just rely on templates: and define the proper conditions to identify the containers properly and start the modules to monitor them.
Hope that makes clear the way autodiscover works in the 2 different approaches (hints-based VS static templates).
Please make sure that hosts hint/config are properly configured so as to be able to reach the services. I see ELASTICSEARCH_HOST which does not look so normal to me, please visit autodsiccover docs to see how hosts/ports are dynamically set in autodiscovered workloads.
With this you should be ok to monitor your services.
Yeah I mean that makes sense, but it doesn't seem to have solved my issue. I've stopped using the hints and instead am using docker.container.name with the templates method, but that doesn't seem to do anything. I'm thinking that somehow I'm using the wrong path to get the name of the containers which is making things go wrong, but I don't know a good way to see how the containers actually look and which fields i should be looking for. The hosts can reach the services, the ELASTICSEARCH_HOST is an environment variable that's needed for the deployment, and works with other elastic services I'm running. I'm just unable to get autodiscover to find my docker stuff. I'm still getting that weird beat.state error I mentioned previously too.
I'm still unsure on which other modules are needed. Do you need the docker module enabled if you're using autodiscover? I'm just trying to get the elasticsearch module to find an elasticsearch instance I'm running in a docker container but nothing is working.
metricbeat.autodiscover:
providers:
- type: docker
labels.dedot: true
templates:
- condition:
contains:
docker.container.name: elasticsearch # make sure that you use the correct name of the container
config:
- module: elasticsearch
metricsets: ["node", "node_stats"]
hosts: "${data.host}:${data.port}"
While trying this, you can run Filebeat in debug mode (filebeat -e -d "*") and check what is logged regarding autodiscovery so as to have a clearer view of what is working and what not.
Thank you! Filebeat in debug mode helped me figure out more what was going on, in addition to which modules were doing what. I also realized that a lot of the info I want was showing up in the metricbeat-* indices however Elasticsearch/Kibana/Linux modules not having a dashboard threw me off. I am still getting the beats state "module not found" error however, and it seems like I get input from my packetbeat container but not from filebeat, however the docker module seems to get info from that container just fine, and it's named as expected. I'm not sure what's up with that
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.