Metricbeat Docker Autodiscover Not Working - ELK

Elastic Stack Beats
1

I have followed the autodiscover docs and my docker container is not being discovered by the service. In order for a container to be discovered, it needs to be labeled, so in order to check docker-compose labels were working, I inspected them and they are indeed present.

My metricbeat.yml:

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config.enabled: false
                         

# setup filebeat to send output to elasticsearch?
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "sample2-%{+YYYY.MM.dd}"

setup.template.name: "sample"
setup.template.pattern: "sample-*"

My docker-compose.yml (myloggingapp):

version: '3.7'
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - discovery.type=single-node
      - cluster.name=docker-
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  filebeat:
    container_name: filebeat
    # needed to override user so we can access the log files, and docker.sock
    user: root
    image: myfilebeat:docker
    volumes:
      # needed to access all docker logs (read only) :
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
      # needed to access additional informations about containers
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - esnet
  metricbeat:
    container_name: metricbeat
    # needed to override user so we can access the log files, and docker.sock
    user: root
    image: mymetricbeat:docker
    volumes:
      # needed to access all docker logs (read only) :
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
      # needed to access additional informations about containers
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - esnet
  myloggingapp:
    container_name: loggingapp
    image: myloggingapp:docker
    labels:
      co.elastic.metrics/enabled: true
      co.elastic.metrics/module: 'docker'
      co.elastic.metrics/metricsets: 'cpu'
      co.elastic.metrics/hosts: 'unix:///var/run/docker.sock'
      co.elastic.metrics/period: '10s'
    networks:
      - esnet      
  elastichq:
    container_name: elastichq
    image: elastichq/elasticsearch-hq
    ports:
      - 8080:5000
    environment:
      - HQ_DEFAULT_URL=http://elasticsearch:9200
      - HQ_ENABLE_SSL=False
      - HQ_DEBUG=FALSE
    networks:
      - esnet  
networks:
  esnet:

The only thing I have read in the logs is a WARN which was:

WARN docker/docker.go:63 BETA: The docker autodiscover is beta

Then the service doesn't seem to go any further than that and hence does not create a metricbeat index. I only get my filebeat index.

What am I doing wrong?

2

I think the problem is the value of the co.elastic.metrics/hosts label in your loggingapp container configuration. I think you want it to be ${data.host}:8080 See the documentation and examples on https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover-hints.html.

Also, you can try running Metricbeat with the logging.level: debug configuration setting to get more details on what autodiscover is doing internally.

3

Thanks for the response, that doesn't make sense as it means everything needs to expose a port, which is not the case for my logging app, as it just logs messages (initially created to test Filebeat). So if your answer is correct, can I just use ${data.host}?

I will also try this too. Can this be set as an env variable like with Logstash?

Also, do I need to mount the containers docker.sock with the host docker.sock in docker-compose?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.