How to use Metricbeat Docker module in Docker Swarm

I'm using Metricbeat in my Docker Swarm and have managed to get the system model pushing CPU and memory metrics to Elastic Search. I can then see the metrics in the sample dashboard in Kibana. However, I can't seem to get the Docker module working. I have enabled it but no logs are shipped and I can't see anything in Kibana for the 'Metricbeat Docker' dashboard. I'm using
the docker.elastic.co/beats/metricbeat:5.6.2 image with Docker 17.06-CE.

This is the metricbeat service in my docker-stack.yml file:

  metricbeat:
    image: my-docker-registry.com/metricbeat:latest
    command: metricbeat -e -system.hostfs=/hostfs
    deploy:
      mode: global # One docker container per node
    environment:
      - MetricbeatHosts=#{MetricbeatHosts}
      - MetricbeatTags=#{MetricbeatTags}
    networks:
      - bridgeoverlay
    volumes:
      - /proc:/hostfs/proc:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /:/hostfs:ro
      - /var/run/docker.sock:/var/run/docker.sock

This is my metricbeat.yml file:

#============================  Config Reloading ===============================

# Config reloading allows to dynamically load modules. Each file which is
# monitored must contain one or multiple modules as a list.
metricbeat.config.modules:

  # Glob pattern for configuration reloading
  path: ${path.config}/conf.d/*.yml

  # Period on which files under path should be checked for chagnes
  reload.period: 10s

  # Set to true to enable config reloading
  reload.enabled: false

#==========================  Modules configuration ============================
metricbeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    # CPU stats
    - cpu

    # System Load stats
    - load

    # Per CPU core stats
    - core

    # IO stats
    - diskio

    # Per filesystem stats
    - filesystem

    # File system summary stats
    - fsstat

    # Memory stats
    - memory

    # Network stats
    - network

    # Per process stats
    - process

    # Sockets and connection info (linux only)
    #- socket
  enabled: true
  period: 10s
  processes: ['.*']

#------------------------------- Docker Module -------------------------------
- module: docker
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  enabled: true
  period: 10s

#-------------------------------- Redis Module -------------------------------
- module: redis
  metricsets: ["info", "keyspace"]
  enabled: true
  period: 10s

  # Redis hosts
  hosts: ["redis"]

  # Timeout after which time a metricset should return an error
  # Timeout is by default defined as period, as a fetch of a metricset
  # should never take longer then period, as otherwise calls can pile up.
  timeout: 1s

#================================ General ======================================

# The tags of the shipper are included in their own field with each
# transaction published. Tags make it easy to group servers by different
# logical properties.
tags: ${MetricbeatTags}

#================================ Outputs ======================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ${MetricbeatHosts}

  # Number of workers per Elasticsearch host.
  worker: 1

  # Optional index name. The default is "metricbeat" plus date
  # and generates [metricbeat-]YYYY.MM.DD keys.
  index: "metricbeat-%{+yyyy.MM.dd}"

  # Configure http request timeout before failing an request to Elasticsearch.
  timeout: 90

  # Path to template file
  template.path: "metricbeat.template.json"

  # Overwrite existing template
  template.overwrite: true
#================================ Logging ======================================
logging.to_files: true
logging.files:

Based on your config I assume you followed the docs here: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html Could you share the log output from metricbeat?

You mention logs above. Do you mean metrics?

Yes, I meant metrics. This is a short extract from my docker service logs:

2017/10/09 04:19:33.305990 metrics.go:39: INFO Non-zero metrics in the last 30s: fetches.docker-container.events=3 fetches.docker-container.failures=3 fetches.docker-cpu.events=3 fetches.docker-cpu.failures=3 fetches.docker-diskio.events=3 fetches.docker-diskio.failures=3 fetches.docker-healthcheck.events=3 fetches.docker-healthcheck.failures=3 fetches.docker-info.events=3 fetches.docker-info.failures=3 fetches.docker-memory.events=3 fetches.docker-memory.failures=3 fetches.docker-network.events=3 fetches.docker-network.failures=3 fetches.redis-info.events=3 fetches.redis-info.success=3 fetches.redis-keyspace.success=3 fetches.system-core.events=6 fetches.system-core.success=3 fetches.system-cpu.events=3 fetches.system-cpu.success=3 fetches.system-diskio.events=6 fetches.system-diskio.success=3 fetches.system-filesystem.events=261 fetches.system-filesystem.success=3 fetches.system-fsstat.events=3 fetches.system-fsstat.success=3 fetches.system-load.events=3 fetches.system-load.success=3 fetches.system-memory.events=3 fetches.system-memory.success=3 fetches.system-network.events=33 fetches.system-network.success=3 fetches.system-process.events=412 fetches.system-process.success=3 libbeat.es.call_count.PublishEvents=17 libbeat.es.publish.read_bytes=10952 libbeat.es.publish.write_bytes=368834 libbeat.es.published_and_acked_events=514 libbeat.publisher.messages_in_worker_queues=754 libbeat.publisher.published_events=754
2017/10/09 04:20:03.306024 metrics.go:39: INFO Non-zero metrics in the last 30s: fetches.docker-container.events=3 fetches.docker-container.failures=3 fetches.docker-cpu.events=3 fetches.docker-cpu.failures=3 fetches.docker-diskio.events=3 fetches.docker-diskio.failures=3 fetches.docker-healthcheck.events=3 fetches.docker-healthcheck.failures=3 fetches.docker-info.events=3 fetches.docker-info.failures=3 fetches.docker-memory.events=3 fetches.docker-memory.failures=3 fetches.docker-network.events=3 fetches.docker-network.failures=3 fetches.redis-info.events=3 fetches.redis-info.success=3 fetches.redis-keyspace.success=3 fetches.system-core.events=6 fetches.system-core.success=3 fetches.system-cpu.events=3 fetches.system-cpu.success=3 fetches.system-diskio.events=6 fetches.system-diskio.success=3 fetches.system-filesystem.events=261 fetches.system-filesystem.success=3 fetches.system-fsstat.events=3 fetches.system-fsstat.success=3 fetches.system-load.events=3 fetches.system-load.success=3 fetches.system-memory.events=3 fetches.system-memory.success=3 fetches.system-network.events=33 fetches.system-network.success=3 fetches.system-process.events=405 fetches.system-process.success=3 libbeat.es.call_count.PublishEvents=39 libbeat.es.publish.read_bytes=25411 libbeat.es.publish.write_bytes=861041 libbeat.es.published_and_acked_events=1248 libbeat.publisher.messages_in_worker_queues=747 libbeat.publisher.published_events=747
2017/10/09 04:20:33.306065 metrics.go:39: INFO Non-zero metrics in the last 30s: fetches.docker-container.events=3 fetches.docker-container.failures=3 fetches.docker-cpu.events=3 fetches.docker-cpu.failures=3 fetches.docker-diskio.events=3 fetches.docker-diskio.failures=3 fetches.docker-healthcheck.events=3 fetches.docker-healthcheck.failures=3 fetches.docker-info.events=3 fetches.docker-info.failures=3 fetches.docker-memory.events=3 fetches.docker-memory.failures=3 fetches.docker-network.events=3 fetches.docker-network.failures=3 fetches.redis-info.events=3 fetches.redis-info.success=3 fetches.redis-keyspace.success=3 fetches.system-core.events=6 fetches.system-core.success=3 fetches.system-cpu.events=3 fetches.system-cpu.success=3 fetches.system-diskio.events=6 fetches.system-diskio.success=3 fetches.system-filesystem.events=261 fetches.system-filesystem.success=3 fetches.system-fsstat.events=3 fetches.system-fsstat.success=3 fetches.system-load.events=3 fetches.system-load.success=3 fetches.system-memory.events=3 fetches.system-memory.success=3 fetches.system-network.events=33 fetches.system-network.success=3 fetches.system-process.events=409 fetches.system-process.success=3 libbeat.es.call_count.PublishEvents=28 libbeat.es.publish.read_bytes=19346 libbeat.es.publish.write_bytes=735968 libbeat.es.published_and_acked_events=1079 libbeat.publisher.messages_in_worker_queues=751 libbeat.publisher.published_events=751
2017/10/09 04:21:03.306066 metrics.go:39: INFO Non-zero metrics in the last 30s: fetches.docker-container.events=3 fetches.docker-container.failures=3 fetches.docker-cpu.events=3 fetches.docker-cpu.failures=3 fetches.docker-diskio.events=3 fetches.docker-diskio.failures=3 fetches.docker-healthcheck.events=3 fetches.docker-healthcheck.failures=3 fetches.docker-info.events=3 fetches.docker-info.failures=3 fetches.docker-memory.events=3 fetches.docker-memory.failures=3 fetches.docker-network.events=3 fetches.docker-network.failures=3 fetches.redis-info.events=3 fetches.redis-info.success=3 fetches.redis-keyspace.success=3 fetches.system-core.events=6 fetches.system-core.success=3 fetches.system-cpu.events=3 fetches.system-cpu.success=3 fetches.system-diskio.events=6 fetches.system-diskio.success=3 fetches.system-filesystem.events=261 fetches.system-filesystem.success=3 fetches.system-fsstat.events=3 fetches.system-fsstat.success=3 fetches.system-load.events=3 fetches.system-load.success=3 fetches.system-memory.events=3 fetches.system-memory.success=3 fetches.system-network.events=33 fetches.system-network.success=3 fetches.system-process.events=405 fetches.system-process.success=3 libbeat.es.call_count.PublishEvents=17 libbeat.es.publish.read_bytes=11743 libbeat.es.publish.write_bytes=506763 libbeat.es.published_and_acked_events=747 libbeat.publisher.messages_in_worker_queues=747 libbeat.publisher.published_events=747

This may be a permission issue while accessing docker socket, do you see any error messages about that? Something you can try is running the container with user: root and see if the problem is gone

I found this other post. It also suggests running as root or using setfacl which I don't really understand. Are those really the only options?

In a nutshell: Metricbeat needs access to the Docker socket to retrieve stats from it, but the docker image we ship runs as a user by default (to make it more secure). There are several ways to allow Metricbeat accessing the docker socket:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.