# Can't get kibana to display filebeat logs with docker auto discovery

**URL:** <https://discuss.elastic.co/t/cant-get-kibana-to-display-filebeat-logs-with-docker-auto-discovery/269810>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [April 11, 2021, 1:53pm UTC](https://discuss.elastic.co/t/cant-get-kibana-to-display-filebeat-logs-with-docker-auto-discovery/269810 "2021-04-11T13:53:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![LongBeachHXC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/longbeachhxc/32/85846_2.png) [@LongBeachHXC](https://discuss.elastic.co/u/LongBeachHXC)\
**Post date:** [April 11, 2021, 1:53pm UTC](https://discuss.elastic.co/t/cant-get-kibana-to-display-filebeat-logs-with-docker-auto-discovery/269810/1 "2021-04-11T13:53:31Z")

</div>

I am trying to leverage the docker auto discovery provider. I followed every tutorial I could find regarding filebeat and I still cannot get Kibana to display the logs. I am trying to get this working with the Elastic Search Containers. I have three ES nodes and one kibana node all running inside docker. I have the filebeat container running as recommended with correct volume mounts and user. It appears data is making it to the Elastic Search cluster because the filebeat index has documents in it. I don't see any log messages in kibana though when I go to logs or discover. Am I missing something in my configuration with filebeat?

I did not notice anything in the filebeat logs to indicate any errors. I see messages like below:

```auto
filebeat | 2021-04-11T13:36:25.953Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/c4e7e39e3f1feb5ec7f6249e5d6b9e1d48455e02f11529888fd01fdef6f24970/c4e7e39e3f1feb5ec7f6249e5d6b9e1d48455e02f11529888fd01fdef6f24970-json.log
filebeat | 2021-04-11T13:36:25.975Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager
filebeat | 2021-04-11T13:36:26.140Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/7cef67c42a0593caca0448f1a534115465b0892670a61547c1aa11c77b8c6d33/*-json.log]
filebeat | 2021-04-11T13:36:26.140Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/7cef67c42a0593caca0448f1a534115465b0892670a61547c1aa11c77b8c6d33/*-json.log]
filebeat | 2021-04-11T13:36:26.140Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/7cef67c42a0593caca0448f1a534115465b0892670a61547c1aa11c77b8c6d33/*-json.log]
filebeat logs
filebeat | 2021-04-11T13:36:27.020Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
filebeat | 2021-04-11T13:36:27.021Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
filebeat | 2021-04-11T13:36:27.021Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
filebeat | 2021-04-11T13:36:27.021Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{filebeat-7.12.0 {now/d}-000001}' as ILM is enabled.
filebeat | 2021-04-11T13:36:27.021Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'filebeat-7.12.0-*' as ILM is enabled.
filebeat | 2021-04-11T13:36:27.021Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.12.0 {now/d}-000001} as ILM is enabled.
filebeat | 2021-04-11T13:36:27.021Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
filebeat | 2021-04-11T13:36:27.027Z INFO template/load.go:97 Template filebeat-7.12.0 already exists and will not be overwritten.
filebeat | 2021-04-11T13:36:27.027Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
filebeat | 2021-04-11T13:36:27.029Z INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
filebeat | 2021-04-11T13:36:27.049Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://es01:9200)) established

```

```auto
filebeat.yml

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

filebeat.modules:

- module: system
- module: elasticsearch
- module: kibana

processors:
  - add_docker_metadata: ~

setup.kibana:
  host: "${CONTAINER_KIBANA}:${PORT_KIBANA}"
  username: "${ELASTIC_KIBANA_SYSTEM_USER}"
  password: "${ELASTIC_KIBANA_SYSTEM_PASSWORD}"
  protocol: "https"
  ssl.enabled: true
  ssl.verification_mode: "certificate"
  ssl.certificate_authorities: 
    - '${CERTS_DIR}/ca/ca.crt'
  ssl.certificate: "${CERTS_DIR}/${CONTAINER_FILEBEAT}/${CONTAINER_FILEBEAT}.crt"
  ssl.key: "${CERTS_DIR}/${CONTAINER_FILEBEAT}/${CONTAINER_FILEBEAT}.key"

output.elasticsearch:
  hosts: ["${CONTAINER_ELASTIC_SEARCH}:${PORT_ELASTIC_SEARCH}"]
  username: "${ELASTIC_BEATS_SYSTEM_USER}"
  password: "${ELASTIC_BEATS_SYSTEM_PASSWORD}"
  protocol: "https"
  ssl.certificate: "${CERTS_DIR}/${CONTAINER_FILEBEAT}/${CONTAINER_FILEBEAT}.crt"
  ssl.key: "${CERTS_DIR}/${CONTAINER_FILEBEAT}/${CONTAINER_FILEBEAT}.key"
  ssl.certificate_authorities: 
    - "${CERTS_DIR}/ca/ca.crt"
  ssl.verification_mode: "certificate"
  ssl.enabled: true

```

```auto
docker-compose.yml

  es01:
    image: ${IMAGE_ELASTIC_SEARCH:?IMAGE_ELASTIC_SEARCH}:${ELASTIC_VERSION:?ELASTIC_VERSION}
    container_name: ${CONTAINER_ELASTIC_SEARCH:?CONTAINER_ELASTIC_SEARCH}01
    labels:
      co.elastic.logs/fileset.stdout: access
      co.elastic.logs/fileset.stderr: error
      co.elastic.logs/module: elasticsearch
    environment:
      - node.name=${CONTAINER_ELASTIC_SEARCH}01
      - cluster.name=${ELASTIC_CLUSTER_NAME:?ELASTIC_CLUSTER_NAME}
      - discovery.seed_hosts=${CONTAINER_ELASTIC_SEARCH}02,${CONTAINER_ELASTIC_SEARCH}03
      - cluster.initial_master_nodes=${CONTAINER_ELASTIC_SEARCH}01,${CONTAINER_ELASTIC_SEARCH}02,${CONTAINER_ELASTIC_SEARCH}03
      - bootstrap.memory_lock=true
      - ELASTIC_PASSWORD=$ELASTIC_BOOTSTRAP_PASSWORD
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=basic
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=${ELASTIC_CERTS_DIR}/${CONTAINER_ELASTIC_SEARCH}01/${CONTAINER_ELASTIC_SEARCH}01.key
      - xpack.security.http.ssl.certificate_authorities=${ELASTIC_CERTS_DIR}/ca/ca.crt
      - xpack.security.http.ssl.certificate=${ELASTIC_CERTS_DIR}/${CONTAINER_ELASTIC_SEARCH}01/${CONTAINER_ELASTIC_SEARCH}01.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=${ELASTIC_CERTS_DIR}/ca/ca.crt
      - xpack.security.transport.ssl.certificate=${ELASTIC_CERTS_DIR}/${CONTAINER_ELASTIC_SEARCH}01/${CONTAINER_ELASTIC_SEARCH}01.crt
      - xpack.security.transport.ssl.key=${ELASTIC_CERTS_DIR}/${CONTAINER_ELASTIC_SEARCH}01/${CONTAINER_ELASTIC_SEARCH}01.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - "${PATH_ROOT}/conf/elk/99-sysctl.conf:/etc/sysctl.d/99-sysctl.conf"
      - data01:/usr/share/elasticsearch/data
      - elastic-certs:${ELASTIC_CERTS_DIR}
      - "${PATH_ROOT}/logs/elk/elastic/:/usr/share/elasticsearch/logs/"
    ports:
      - 9200:9200
    networks:
      dotmusic_net:
        ipv4_address: ${DOCKER_IP_ELASTIC_1:?DOCKER_IP_ELASTIC_1}
    healthcheck:
      test: curl --cacert ${ELASTIC_CERTS_DIR}/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[$$? == 52]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  kibana:
    image: ${IMAGE_KIBANA:?IMAGE_KIBANA}
    container_name: ${CONTAINER_KIBANA}
    environment:
      SERVER_NAME: ${KIBANA_SERVER_NAME}
      ELASTIC_SEARCH_URL: https://${CONTAINER_ELASTIC_SEARCH}01:9200
      ELASTICSEARCH_HOSTS: '["https://${CONTAINER_ELASTIC_SEARCH}01:9200","https://${CONTAINER_ELASTIC_SEARCH}02:9200","https://${CONTAINER_ELASTIC_SEARCH}03:9200"]'
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_CERTIFICATE: ${ELASTIC_CERTS_DIR}/${CONTAINER_KIBANA}/${CONTAINER_KIBANA}.crt
      SERVER_SSL_KEY: ${ELASTIC_CERTS_DIR}/${CONTAINER_KIBANA}/${CONTAINER_KIBANA}.key
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: '["${ELASTIC_CERTS_DIR}/ca/ca.crt"]'
      ELASTICSEARCH_USERNAME: ${ELASTIC_KIBANA_SYSTEM_USER:?ELASTIC_KIBANA_SYSTEM_USER}
      ELASTICSEARCH_PASSWORD: ${ELASTIC_KIBANA_SYSTEM_PASSWORD:?ELASTIC_KIBANA_SYSTEM_PASSWORD}
      XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: ${KIBANA_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY:?KIBANA_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY}
    labels:
      co.elastic.logs/enabled: "true"
      co.elastic.logs/module: "kibana"
      co.elastic.logs/fileset.stdout: access
      co.elastic.logs/fileset.stderr: error
    ports:
      - 5601:5601
    volumes:
      - elastic-certs:${ELASTIC_CERTS_DIR}
    networks:
      dotmusic_net:
        ipv4_address: ${DOCKER_IP_KIBANA:?DOCKER_IP_KIBANA}

  filebeat:
    image: ${IMAGE_FILEBEAT:?IMAGE_FILEBEAT}:${FILEBEAT_VERSION:?FILEBEAT_VERSION}
    container_name: ${CONTAINER_FILEBEAT:?CONTAINER_FILEBEAT}
    user: root
    volumes:
      - "${PATH_ROOT}/conf/elk/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - elastic-certs:${ELASTIC_CERTS_DIR}
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
    depends_on: 
      - ${CONTAINER_ELASTIC_SEARCH}01
      - ${CONTAINER_KIBANA}
    environment:
      CONTAINER_ELASTIC_SEARCH: ${CONTAINER_ELASTIC_SEARCH}01
      PORT_ELASTIC_SEARCH: ${PORT_ELASTIC_SEARCH}
      CONTAINER_KIBANA: ${CONTAINER_KIBANA}
      PORT_KIBANA: ${PORT_KIBANA}
      CERTS_DIR: ${ELASTIC_CERTS_DIR}
      CONTAINER_FILEBEAT: ${CONTAINER_FILEBEAT}
      ELASTIC_BEATS_SYSTEM_USER: ${ELASTIC_USER}
      ELASTIC_BEATS_SYSTEM_PASSWORD: ${ELASTIC_PASSWORD}
      ELASTIC_KIBANA_SYSTEM_USER: ${ELASTIC_USER}
      ELASTIC_KIBANA_SYSTEM_PASSWORD: ${ELASTIC_PASSWORD}
    networks:
      dotmusic_net:
        ipv4_address: ${DOCKER_IP_FILEBEAT:?DOCKER_IP_FILEBEAT}

```

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [April 14, 2021, 12:03am UTC](https://discuss.elastic.co/t/cant-get-kibana-to-display-filebeat-logs-with-docker-auto-discovery/269810/2 "2021-04-14T00:03:59Z")

</div>

Hey @LongBeachHXC 🙂

That seems like a weird issue indeed, especially if you mention that there are data in the index, but discover is unable to view it, you are not logged in using a user that does not have access to view the data? Or using document level security or something similar right?

Could you give the Kibana Dev tools a try, and run a blank search on the index, and maybe past in one of the documents? If search does not return anything either it does sound like a much bigger problem, and at that point it might be time to look into the ES or Kibana logs.

The API call should just need to be something like:  
`GET /filebeat/_search`

---

<div class="post-metadata">

**Author:** ![LongBeachHXC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/longbeachhxc/32/85846_2.png) [@LongBeachHXC](https://discuss.elastic.co/u/LongBeachHXC)\
**Post date:** [April 17, 2021, 2:09pm UTC](https://discuss.elastic.co/t/cant-get-kibana-to-display-filebeat-logs-with-docker-auto-discovery/269810/3 "2021-04-17T14:09:13Z")

</div>

So, now my filebeat isn't creating and pushing to a new index. I expect to see a new index in stack management for filebeat with today's date but I don't see anything. Nothing in the filebeat logs indicate an error. I am just perplexed, I've been trying to get this filebeat working for days now and still cannot get it to work.

```auto
filebeat logs

filebeat | 2021-04-17T13:57:28.612Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
filebeat | 2021-04-17T13:57:28.612Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
filebeat | 2021-04-17T13:57:28.613Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
filebeat | 2021-04-17T13:57:28.613Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{filebeat-7.12.0 {now/d}-000001}' as ILM is enabled.
filebeat | 2021-04-17T13:57:28.613Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'filebeat-7.12.0-*' as ILM is enabled.
filebeat | 2021-04-17T13:57:28.613Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.12.0 {now/d}-000001} as ILM is enabled.
filebeat | 2021-04-17T13:57:28.613Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
filebeat | 2021-04-17T13:57:28.614Z INFO template/load.go:97 Template filebeat-7.12.0 already exists and will not be overwritten.
filebeat | 2021-04-17T13:57:28.614Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
filebeat | 2021-04-17T13:57:28.615Z INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
filebeat | 2021-04-17T13:57:28.615Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://es01:9200)) established

```

---

<div class="post-metadata">

**Author:** ![LongBeachHXC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/longbeachhxc/32/85846_2.png) [@LongBeachHXC](https://discuss.elastic.co/u/LongBeachHXC)\
**Post date:** [April 17, 2021, 2:11pm UTC](https://discuss.elastic.co/t/cant-get-kibana-to-display-filebeat-logs-with-docker-auto-discovery/269810/4 "2021-04-17T14:11:39Z")

</div>

Here is some output from my dev tools

```auto
GET _cat/indices/filebeat*

green open filebeat-7.12.0-2021.04.10-000001 PThT86iqRlqAuVF2f3r4qw 1 1 8706 0 2.8mb 1.4mb

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 15, 2021, 4:12pm UTC](https://discuss.elastic.co/t/cant-get-kibana-to-display-filebeat-logs-with-docker-auto-discovery/269810/5 "2021-05-15T16:12:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.