How to use Metricbeat Docker module in Docker Swarm

Elastic Stack Beats
1

I'm using Metricbeat in my Docker Swarm and have managed to get the system model pushing CPU and memory metrics to Elastic Search. I can then see the metrics in the sample dashboard in Kibana. However, I can't seem to get the Docker module working. I have enabled it but no logs are shipped and I can't see anything in Kibana for the 'Metricbeat Docker' dashboard. I'm using
the docker.elastic.co/beats/metricbeat:5.6.2 image with Docker 17.06-CE.

This is the metricbeat service in my docker-stack.yml file:

  metricbeat:
    image: my-docker-registry.com/metricbeat:latest
    command: metricbeat -e -system.hostfs=/hostfs
    deploy:
      mode: global # One docker container per node
    environment:
      - MetricbeatHosts=#{MetricbeatHosts}
      - MetricbeatTags=#{MetricbeatTags}
    networks:
      - bridgeoverlay
    volumes:
      - /proc:/hostfs/proc:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /:/hostfs:ro
      - /var/run/docker.sock:/var/run/docker.sock

This is my metricbeat.yml file:

#============================  Config Reloading ===============================

# Config reloading allows to dynamically load modules. Each file which is
# monitored must contain one or multiple modules as a list.
metricbeat.config.modules:

  # Glob pattern for configuration reloading
  path: ${path.config}/conf.d/*.yml

  # Period on which files under path should be checked for chagnes
  reload.period: 10s

  # Set to true to enable config reloading
  reload.enabled: false

#==========================  Modules configuration ============================
metricbeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    # CPU stats
    - cpu

    # System Load stats
    - load

    # Per CPU core stats
    - core

    # IO stats
    - diskio

    # Per filesystem stats
    - filesystem

    # File system summary stats
    - fsstat

    # Memory stats
    - memory

    # Network stats
    - network

    # Per process stats
    - process

    # Sockets and connection info (linux only)
    #- socket
  enabled: true
  period: 10s
  processes: ['.*']

#------------------------------- Docker Module -------------------------------
- module: docker
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  enabled: true
  period: 10s

#-------------------------------- Redis Module -------------------------------
- module: redis
  metricsets: ["info", "keyspace"]
  enabled: true
  period: 10s

  # Redis hosts
  hosts: ["redis"]

  # Timeout after which time a metricset should return an error
  # Timeout is by default defined as period, as a fetch of a metricset
  # should never take longer then period, as otherwise calls can pile up.
  timeout: 1s

#================================ General ======================================

# The tags of the shipper are included in their own field with each
# transaction published. Tags make it easy to group servers by different
# logical properties.
tags: ${MetricbeatTags}

#================================ Outputs ======================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ${MetricbeatHosts}

  # Number of workers per Elasticsearch host.
  worker: 1

  # Optional index name. The default is "metricbeat" plus date
  # and generates [metricbeat-]YYYY.MM.DD keys.
  index: "metricbeat-%{+yyyy.MM.dd}"

  # Configure http request timeout before failing an request to Elasticsearch.
  timeout: 90

  # Path to template file
  template.path: "metricbeat.template.json"

  # Overwrite existing template
  template.overwrite: true
#================================ Logging ======================================
logging.to_files: true
logging.files:
2

Based on your config I assume you followed the docs here: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html Could you share the log output from metricbeat?

You mention logs above. Do you mean metrics?

3

Yes, I meant metrics. This is a short extract from my docker service logs:

2017/10/09 04:19:33.305990 metrics.go:39: INFO Non-zero metrics in the last 30s: fetches.docker-container.events=3 fetches.docker-container.failures=3 fetches.docker-cpu.events=3 fetches.docker-cpu.failures=3 fetches.docker-diskio.events=3 fetches.docker-diskio.failures=3 fetches.docker-healthcheck.events=3 fetches.docker-healthcheck.failures=3 fetches.docker-info.events=3 fetches.docker-info.failures=3 fetches.docker-memory.events=3 fetches.docker-memory.failures=3 fetches.docker-network.events=3 fetches.docker-network.failures=3 fetches.redis-info.events=3 fetches.redis-info.success=3 fetches.redis-keyspace.success=3 fetches.system-core.events=6 fetches.system-core.success=3 fetches.system-cpu.events=3 fetches.system-cpu.success=3 fetches.system-diskio.events=6 fetches.system-diskio.success=3 fetches.system-filesystem.events=261 fetches.system-filesystem.success=3 fetches.system-fsstat.events=3 fetches.system-fsstat.success=3 fetches.system-load.events=3 fetches.system-load.success=3 fetches.system-memory.events=3 fetches.system-memory.success=3 fetches.system-network.events=33 fetches.system-network.success=3 fetches.system-process.events=412 fetches.system-process.success=3 libbeat.es.call_count.PublishEvents=17 libbeat.es.publish.read_bytes=10952 libbeat.es.publish.write_bytes=368834 libbeat.es.published_and_acked_events=514 libbeat.publisher.messages_in_worker_queues=754 libbeat.publisher.published_events=754
2017/10/09 04:20:03.306024 metrics.go:39: INFO Non-zero metrics in the last 30s: fetches.docker-container.events=3 fetches.docker-container.failures=3 fetches.docker-cpu.events=3 fetches.docker-cpu.failures=3 fetches.docker-diskio.events=3 fetches.docker-diskio.failures=3 fetches.docker-healthcheck.events=3 fetches.docker-healthcheck.failures=3 fetches.docker-info.events=3 fetches.docker-info.failures=3 fetches.docker-memory.events=3 fetches.docker-memory.failures=3 fetches.docker-network.events=3 fetches.docker-network.failures=3 fetches.redis-info.events=3 fetches.redis-info.success=3 fetches.redis-keyspace.success=3 fetches.system-core.events=6 fetches.system-core.success=3 fetches.system-cpu.events=3 fetches.system-cpu.success=3 fetches.system-diskio.events=6 fetches.system-diskio.success=3 fetches.system-filesystem.events=261 fetches.system-filesystem.success=3 fetches.system-fsstat.events=3 fetches.system-fsstat.success=3 fetches.system-load.events=3 fetches.system-load.success=3 fetches.system-memory.events=3 fetches.system-memory.success=3 fetches.system-network.events=33 fetches.system-network.success=3 fetches.system-process.events=405 fetches.system-process.success=3 libbeat.es.call_count.PublishEvents=39 libbeat.es.publish.read_bytes=25411 libbeat.es.publish.write_bytes=861041 libbeat.es.published_and_acked_events=1248 libbeat.publisher.messages_in_worker_queues=747 libbeat.publisher.published_events=747
2017/10/09 04:20:33.306065 metrics.go:39: INFO Non-zero metrics in the last 30s: fetches.docker-container.events=3 fetches.docker-container.failures=3 fetches.docker-cpu.events=3 fetches.docker-cpu.failures=3 fetches.docker-diskio.events=3 fetches.docker-diskio.failures=3 fetches.docker-healthcheck.events=3 fetches.docker-healthcheck.failures=3 fetches.docker-info.events=3 fetches.docker-info.failures=3 fetches.docker-memory.events=3 fetches.docker-memory.failures=3 fetches.docker-network.events=3 fetches.docker-network.failures=3 fetches.redis-info.events=3 fetches.redis-info.success=3 fetches.redis-keyspace.success=3 fetches.system-core.events=6 fetches.system-core.success=3 fetches.system-cpu.events=3 fetches.system-cpu.success=3 fetches.system-diskio.events=6 fetches.system-diskio.success=3 fetches.system-filesystem.events=261 fetches.system-filesystem.success=3 fetches.system-fsstat.events=3 fetches.system-fsstat.success=3 fetches.system-load.events=3 fetches.system-load.success=3 fetches.system-memory.events=3 fetches.system-memory.success=3 fetches.system-network.events=33 fetches.system-network.success=3 fetches.system-process.events=409 fetches.system-process.success=3 libbeat.es.call_count.PublishEvents=28 libbeat.es.publish.read_bytes=19346 libbeat.es.publish.write_bytes=735968 libbeat.es.published_and_acked_events=1079 libbeat.publisher.messages_in_worker_queues=751 libbeat.publisher.published_events=751
2017/10/09 04:21:03.306066 metrics.go:39: INFO Non-zero metrics in the last 30s: fetches.docker-container.events=3 fetches.docker-container.failures=3 fetches.docker-cpu.events=3 fetches.docker-cpu.failures=3 fetches.docker-diskio.events=3 fetches.docker-diskio.failures=3 fetches.docker-healthcheck.events=3 fetches.docker-healthcheck.failures=3 fetches.docker-info.events=3 fetches.docker-info.failures=3 fetches.docker-memory.events=3 fetches.docker-memory.failures=3 fetches.docker-network.events=3 fetches.docker-network.failures=3 fetches.redis-info.events=3 fetches.redis-info.success=3 fetches.redis-keyspace.success=3 fetches.system-core.events=6 fetches.system-core.success=3 fetches.system-cpu.events=3 fetches.system-cpu.success=3 fetches.system-diskio.events=6 fetches.system-diskio.success=3 fetches.system-filesystem.events=261 fetches.system-filesystem.success=3 fetches.system-fsstat.events=3 fetches.system-fsstat.success=3 fetches.system-load.events=3 fetches.system-load.success=3 fetches.system-memory.events=3 fetches.system-memory.success=3 fetches.system-network.events=33 fetches.system-network.success=3 fetches.system-process.events=405 fetches.system-process.success=3 libbeat.es.call_count.PublishEvents=17 libbeat.es.publish.read_bytes=11743 libbeat.es.publish.write_bytes=506763 libbeat.es.published_and_acked_events=747 libbeat.publisher.messages_in_worker_queues=747 libbeat.publisher.published_events=747
4

This may be a permission issue while accessing docker socket, do you see any error messages about that? Something you can try is running the container with user: root and see if the problem is gone

5

I found this other post. It also suggests running as root or using setfacl which I don't really understand. Are those really the only options?

6

In a nutshell: Metricbeat needs access to the Docker socket to retrieve stats from it, but the docker image we ship runs as a user by default (to make it more secure). There are several ways to allow Metricbeat accessing the docker socket:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.