Metricbeats monitoring elasticsearch in docker: elasticsearch module or autodiscover?

[sgreszcz (Stephen Greszczyszyn)]

Hi there,

I'm working on metricbeats 6.3 to monitor several applications in docker containers where one of them contains the official elasticsearch docker image.

I'm using this syntax for docker:

metricbeat.modules:
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-docker.html
- module: docker
  enabled: true
  period: 10s
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]

But I'm not sure how to do the elasticsearch monitoring - should that be with the docker metricbeat.autodiscover feature or the elasticsearch module. Also is the metricbeat.autodiscover redundant if I'm already using "metricbeat.modules: -docker"?

For elasticsearch:

#------------------------------- Docker Monitoring -------------------------------
metricbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: elasticsearch
          config:
            - module: elasticsearch
              metricsets: ["node", "nodestats"]
              hosts: "${data.host}:{data.port}"

or

#---------------------------- Elasticsearch Module ---------------------------
metricbeat.modules:
- module: elasticsearch
  metricsets: ["node", "node_stats"]
  period: 10s
  hosts: ["localhost:9200"]

Thank you for any guidance!

[jsoriano (Jaime Soriano)]

Hi @sgreszcz,

The use of autodiscover depends on how you deploy your applications. If you know in what nodes your applications are going to run, then you can use the module directly. If the selection of the node depends on some kind of orchestrator, then you need to make use of autodiscover. Autodiscover can be also useful in the case you control where the applications run, for example if you want to provide a common configuration for all your nodes, or if you want to deploy the configuration before the application is running.

The docker module is not redundant with autodiscover, this module collects general metrics of containers running in docker, as resources usage, but they don't collect the specific metrics of the service running in the container, this is the job of modules configured with autodiscover. In general it is a good idea to use both, the docker module, and autodiscover for metrics of specific containers.