How to Run Metricbeat in Docker with Modules to Monitor Other Containers

Hello all, I'm trying to use Metricbeat to monitor my elasticstack. Logstash, Elasticsearch, Kibana, and some Beats are all running inside of docker containers. I am wondering what is the best way to use Metricbeat to monitor these processes?
I believe I have mounted the filesystems correctly and am running Metricbeat with the correct options, however I am unsure how to combine the Hints Based Autodiscover with the Modules and Dynamic Config Reloading.
Do I need to define my modules in the modules.d directory? Do I need to define them as Templates under Hint Based Autodiscover? Will it work if I simply define them as Modules in the metricbeat.yml configuration file? Will these work with the Dynamic Config Reloading?
Here is my configuration for metricbeat.yml

########################## Metricbeat Configuration ###########################

# This file is a full configuration example documenting all non-deprecated
# options in comments. For a shorter configuration example, that contains only
# the most common options, please see metricbeat.yml in the same directory.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#============================  Config Reloading ===============================
# Config reloading allows to dynamically load modules. Each file which is
# monitored must contain one or multiple modules as a list.
metricbeat.config.modules:
  # Glob pattern for configuration reloading
  path: ${path.config}/modules.d/*.yml
  # Period on which files under path should be checked for changes
  reload.period: 10s
  # Set to true to enable config reloading
  reload.enabled: true

# Maximum amount of time to randomly delay the start of a metricset. Use 0 to
# disable startup delay.
metricbeat.max_start_delay: 10s

#============================== Autodiscover ===================================
# Autodiscover allows you to detect changes in the system and spawn new modules
# as they happen.
metricbeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: docker
      hints.enabled: true

#==========================  Modules configuration =============================
metricbeat.modules:
#-------------------------------- System Module --------------------------------
- module: system
  metricsets:
    - cpu             # CPU usage
    - load            # CPU load averages
    - memory          # Memory usage
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - uptime          # System Uptime
    - socket_summary  # Socket summary
    #- core           # Per CPU core usage
    #- diskio         # Disk IO
    #- filesystem     # File system usage for each mountpoint
    #- fsstat         # File system summary metrics
    #- raid           # Raid
    #- socket         # Sockets and connection info (linux only)
    - service        # systemd service information
  enabled: true
  period: 10s
  processes: ['.*']

  # Configure the metric types that are included by these metricsets.
  cpu.metrics:  ["percentages","normalized_percentages"]  # The other available option is ticks.
  core.metrics: ["percentages"]  # The other available option is ticks.

  # A list of filesystem types to ignore. 
  #filesystem.ignore_types: []

  process.cgroups.enabled: true


#--------------------------------- Beat Module ---------------------------------
- module: beat
  metricsets:
    - stats
    - state
  period: 10s
  hosts: ["http://localhost:5066"]
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  #ssl.certificate: "/etc/pki/client/cert.pem"
  #ssl.key: "/etc/pki/client/cert.key"
  # Set to true to send data collected by module to X-Pack
  # Monitoring instead of metricbeat-* indices.
  #xpack.enabled: false

#-------------------------------- Docker Module --------------------------------
- module: docker
  metricsets:
    - "container"
    - "cpu"
    - "diskio"
    - "event"
    - "healthcheck"
    - "info"
    #- "image"
    - "memory"
    - "network"
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

  # If set to true, replace dots in labels with `_`.
  #labels.dedot: false

  # If set to true, collects metrics per core.
  #cpu.cores: true

  # To connect to Docker over TLS you must specify a client and CA certificate.
  #ssl:
  #  certificate_authority: "/etc/pki/root/ca.pem"
  #  certificate:           "/etc/pki/client/cert.pem"
  #  key:                   "/etc/pki/client/cert.key"

#---------------------------- Elasticsearch Module ----------------------------
- module: elasticsearch
  metricsets:
    - node
    - node_stats
    #- index
    #- index_recovery
    #- index_summary
    #- shard
    #- ml_job
  period: 10s
  hosts: ["${ELASTICSEARCH_HOST}"]
  username: ${ES_USER}
  password: ${ES_PASSWORD}
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Set to false to fetch all entries
  #index_recovery.active_only: true

  # Set to true to send data collected by module to X-Pack
  # Monitoring instead of metricbeat-* indices.
  xpack.enabled: false

#-------------------------------- Kibana Module --------------------------------
- module: kibana
  metricsets: ["status"]
  period: 10s
  hosts: ["${KIBANA_HOST}"]
  username: "${ES_USER}"
  password: "${ES_PASSWORD}"
  basepath: ""
  enabled: true
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"
  # Set to true to send data collected by module to X-Pack
  # Monitoring instead of metricbeat-* indices.
  xpack.enabled: false

#-------------------------------- Linux Module --------------------------------
- module: linux
  period: 10s
  metricsets:
    - "pageinfo"
    # - ksm
    # - conntrack
  enabled: true
  hostfs: /hostfs

#------------------------------- Logstash Module -------------------------------
- module: logstash
  metricsets: ["node", "node_stats"]
  enabled: true
  period: 10s
  hosts: ["${LOGSTASH_HOST2}"]

# ================================== General ===================================
# The following example enriches each event with docker metadata, it matches
# container id from log path available in `source` field (by default it expects
# it to be /var/lib/docker/containers/*/*.log).
processors:
  - add_docker_metadata: ~

# The following example enriches each event with host metadata.
processors:
  - add_host_metadata: ~

# The following example enriches each event with process metadata using
# process IDs included in the event.
processors:
  - add_process_metadata:
      match_pids: ["system.process.ppid"]
      target: system.process.parent

# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  enabled: false
  hosts: ["${ELASTICSEARCH_HOST}"]
  username: ${ES_USER}
  password: ${ES_PASSWORD}
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # Boolean flag to enable or disable the output module.
  enabled: true
  hosts: ["${LOGSTASH_HOST}"]
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  host: "${KIBANA_HOST}"
  username: ${ES_USER}
  password: ${ES_PASSWORD}
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key

When I check Kibana Metrics, I can see only 1 host (Metricbeat). When I check the docker containers, all of my containers do appear but it appears like it isn't applying any modules to them.

Here is my docker compose config for metricbeat. For the other services I also have a "labels" section where I define the module but that's it.

metricbeat:
    image: docker.elastic.co/beats/metricbeat:${ELASTIC_VERSION}
    hostname: metricbeat
    depends_on:
      - kibana
      - elasticsearch 
    restart: on-failure
    command: metricbeat -e -system.hostfs=/hostfs
    user: "0"
    cap_add:
      - SYS_PTRACE
      - DAC_READ_SEARCH         
      #- CAP_SYS_ADMIN
    network_mode: host
    volumes:
      - ./metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /proc:/hostfs/proc:ro
      - /:/hostfs:ro
      - ./generated/certs/root.pem:/etc/pki/root/ca.pem:ro
      - ./generated/certs/elastic-chain.pem:/etc/pki/client/cert.pem:ro
      - ./generated/certs/elastic-key.pem:/etc/pki/client/cert.key:ro
    environment:
 ...

In addition I am getting this error in my logs:

metricbeat_1             | 2020-09-01T16:34:04.396Z	INFO	module/wrapper.go:259	Error fetching data for metricset beat.state: failure to apply state schema: 1 error: key `module` not found

I'm not sure what I'm doing wrong in my configurations, and help is greatly appreciated!

Hi!

You can use Hints based autodiscover to monitor the services that run as containers (kibana, logstash etc). Note that you need to annotate these containers with the proper labels so as to make autodiscover able to identify how you want to monitor them (with the respective Metricbeat module). In this please have a look at the respective docs: https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover-hints.html#_docker_3

Otherwise if you know the targeted services beforehand you can just use the proper templates so as to define static autodiscover rules before starting Metricbeat. See https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html

With autodiscover you should be able to launch the proper modules to monitor the services that run as containers. Let me know if you have any questions about these!

C.

Hi Chris, thank you so much for your reply, your help is really appreciated.

Okay so I've read the autodiscover docs and added some labels to my other containers eg:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
    hostname: elasticsearch
    labels:
      - "co.elastic.metrics/module=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
...

I know that that label is for the "hint based" setting but I hope that this is a good hybrid so I can identify which services are in which container even if I do a custom image for them.
My new metricbeat config is:

########################## Metricbeat Configuration ###########################

# This file is a full configuration example documenting all non-deprecated
# options in comments. For a shorter configuration example, that contains only
# the most common options, please see metricbeat.yml in the same directory.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#============================  Config Reloading ===============================
# Config reloading allows to dynamically load modules. Each file which is
# monitored must contain one or multiple modules as a list.
metricbeat.config.modules:
  # Glob pattern for configuration reloading
  path: ${path.config}/modules.d/*.yml
  # Period on which files under path should be checked for changes
  reload.period: 10s
  # Set to true to enable config reloading
  reload.enabled: false

# Maximum amount of time to randomly delay the start of a metricset. Use 0 to
# disable startup delay.
metricbeat.max_start_delay: 10s

#============================== Autodiscover ===================================
# Autodiscover allows you to detect changes in the system and spawn new modules
# as they happen.
metricbeat.autodiscover:
  # List of enabled autodiscover providers
  providers:
    - type: docker
      hints.enabled: false
      templates:
#---------------------------- Elasticsearch Module ----------------------------
        - condition:
            contains:
              docker.container.labels.co.elastic.metrics/module: "elasticsearch"
          config:
            - module: elasticsearch
              metricsets:
                - node
                - node_stats
                #- index
                #- index_recovery
                #- index_summary
                #- shard
                #- ml_job
              period: 10s
              hosts: ["${ELASTICSEARCH_HOST}"]
              username: ${ES_USER}
              password: ${ES_PASSWORD}
              ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
              xpack.enabled: false
#--------------------------------- Beat Module ---------------------------------
        - condition:
            contains:
              docker.container.labels.co.elastic.metrics/module: "beat"
          config:
            - module: beat
              metricsets:
                - stats
                - state
              period: 10s
              hosts: ["http://localhost:5066"]
              #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
              #ssl.certificate: "/etc/pki/client/cert.pem"
              #ssl.key: "/etc/pki/client/cert.key"
              # Set to true to send data collected by module to X-Pack
              # Monitoring instead of metricbeat-* indices.
              #xpack.enabled: false
#-------------------------------- Kibana Module --------------------------------
        - condition:
            contains:
              docker.container.labels.co.elastic.metrics/module: "kibana"
          config:
            - module: kibana
              metricsets: ["status"]
              period: 10s
              hosts: ["${KIBANA_HOST}"]
              username: "${ES_USER}"
              password: "${ES_PASSWORD}"
              basepath: ""
              enabled: true
              ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
              ssl.certificate: "/etc/pki/client/cert.pem"
              ssl.key: "/etc/pki/client/cert.key"
              # Set to true to send data collected by module to X-Pack
              # Monitoring instead of metricbeat-* indices.
              xpack.enabled: false
#------------------------------- Logstash Module -------------------------------
        - condition:
            contains:
              docker.container.labels.co.elastic.metrics/module: "logstash"
          config:
            - module: logstash
              metricsets: ["node", "node_stats"]
              enabled: true
              period: 10s
              hosts: ["${LOGSTASH_HOST2}"]


#==========================  Modules configuration =============================
metricbeat.modules:
#-------------------------------- System Module --------------------------------
- module: system
  metricsets:
    - cpu             # CPU usage
    - load            # CPU load averages
    - memory          # Memory usage
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - uptime          # System Uptime
    - socket_summary  # Socket summary
    #- core           # Per CPU core usage
    #- diskio         # Disk IO
    #- filesystem     # File system usage for each mountpoint
    #- fsstat         # File system summary metrics
    #- raid           # Raid
    #- socket         # Sockets and connection info (linux only)
    - service        # systemd service information
  enabled: true
  period: 10s
  processes: ['.*']

  # Configure the metric types that are included by these metricsets.
  cpu.metrics:  ["percentages","normalized_percentages"]  # The other available option is ticks.
  core.metrics: ["percentages"]  # The other available option is ticks.

  # A list of filesystem types to ignore. 
  #filesystem.ignore_types: []

  process.cgroups.enabled: true

  # Configure reverse DNS lookup on remote IP addresses in the socket metricset.
  #socket.reverse_lookup.enabled: false
  #socket.reverse_lookup.success_ttl: 60s
  #socket.reverse_lookup.failure_ttl: 60s

  # Filter systemd services by status or sub-status
  #service.state_filter: ["active"]

  # Filter systemd services based on a name pattern
  #service.pattern_filter: ["ssh*", "nfs*"]


#-------------------------------- Docker Module --------------------------------
- module: docker
  metricsets:
    - "container"
    - "cpu"
    - "diskio"
    - "event"
    - "healthcheck"
    - "info"
    #- "image"
    - "memory"
    - "network"
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

  # If set to true, replace dots in labels with `_`.
  #labels.dedot: false

  # If set to true, collects metrics per core.
  #cpu.cores: true

  # To connect to Docker over TLS you must specify a client and CA certificate.
  #ssl:
  #  certificate_authority: "/etc/pki/root/ca.pem"
  #  certificate:           "/etc/pki/client/cert.pem"
  #  key:                   "/etc/pki/client/cert.key"


#-------------------------------- Linux Module --------------------------------
- module: linux
  period: 10s
  metricsets:
    - "pageinfo"
    # - ksm
    # - conntrack
  enabled: true
  hostfs: /hostfs


# ================================== General ===================================
# The following example enriches each event with docker metadata, it matches
# container id from log path available in `source` field (by default it expects
# it to be /var/lib/docker/containers/*/*.log).
processors:
  - add_docker_metadata: ~

# The following example enriches each event with host metadata.
processors:
  - add_host_metadata: ~

# The following example enriches each event with process metadata using
# process IDs included in the event.
processors:
  - add_process_metadata:
      match_pids: ["system.process.ppid"]
      target: system.process.parent

# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  enabled: false
  hosts: ["${ELASTICSEARCH_HOST}"]
  username: ${ES_USER}
  password: ${ES_PASSWORD}
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # Boolean flag to enable or disable the output module.
  enabled: true
  hosts: ["${LOGSTASH_HOST}"]
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  host: "${KIBANA_HOST}"
  username: ${ES_USER}
  password: ${ES_PASSWORD}
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"

Does this look correct to you? I'm not sure which modules are still needed to be defined outside of it or if the linux/docker modules are needed, or if I need to define the modules for the things I want autodiscovered outside of the "template" part of the autodiscover. I am still getting this strange error in my logs: INFO module/wrapper.go:259 Error fetching data for metricset beat.state: failure to apply state schema: 1 error: key 'module' not found, in addition still only the "Metricbeat System" host shows up when I look under the Kibana Metricbeat System ECS dashboard. Same for "Containers Overview" dashboard, it appears that "Metricbeat System" is the only one that appears. I'm not sure if this is normal or showing there's something misconfigured. Under "Metrics", Metricbeat shows up as the only host but the other containers are visible, but they don't show up the same way that Elasticsearch would if I was monitoring it outside of a container.
Thank you for the help!

Hi again!

You should not mix hints/annotations with static templates. If you want to annotate the containers and make metricbeat identify them and start monitoring them with the proper modules you need to enable hints-based autodiscover:

hints.enabled: true

Then Metricbeat will be able to catch the conteiners' labels and start monitoring them based on their labels. So templates: should be removed.

Otherwise you can just rely on templates: and define the proper conditions to identify the containers properly and start the modules to monitor them.

Hope that makes clear the way autodiscover works in the 2 different approaches (hints-based VS static templates).

Please make sure that hosts hint/config are properly configured so as to be able to reach the services. I see ELASTICSEARCH_HOST which does not look so normal to me, please visit autodsiccover docs to see how hosts/ports are dynamically set in autodiscovered workloads.

With this you should be ok to monitor your services.

Yeah I mean that makes sense, but it doesn't seem to have solved my issue. I've stopped using the hints and instead am using docker.container.name with the templates method, but that doesn't seem to do anything. I'm thinking that somehow I'm using the wrong path to get the name of the containers which is making things go wrong, but I don't know a good way to see how the containers actually look and which fields i should be looking for. The hosts can reach the services, the ELASTICSEARCH_HOST is an environment variable that's needed for the deployment, and works with other elastic services I'm running. I'm just unable to get autodiscover to find my docker stuff. I'm still getting that weird beat.state error I mentioned previously too.

I'm still unsure on which other modules are needed. Do you need the docker module enabled if you're using autodiscover? I'm just trying to get the elasticsearch module to find an elasticsearch instance I'm running in a docker container but nothing is working.

Hi!

Something like this below should work:

metricbeat.autodiscover:
  providers:
    - type: docker
      labels.dedot: true
      templates:
        - condition:
            contains:
              docker.container.name: elasticsearch # make sure that you use the correct name of the container
          config:
            - module: elasticsearch
              metricsets: ["node", "node_stats"]
              hosts: "${data.host}:${data.port}"

While trying this, you can run Filebeat in debug mode (filebeat -e -d "*") and check what is logged regarding autodiscovery so as to have a clearer view of what is working and what not.

Thank you! Filebeat in debug mode helped me figure out more what was going on, in addition to which modules were doing what. I also realized that a lot of the info I want was showing up in the metricbeat-* indices however Elasticsearch/Kibana/Linux modules not having a dashboard threw me off. I am still getting the beats state "module not found" error however, and it seems like I get input from my packetbeat container but not from filebeat, however the docker module seems to get info from that container just fine, and it's named as expected. I'm not sure what's up with that