# How to collect containerized elasticsearch logs with filebeat

**URL:** <https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [March 8, 2022, 4:41pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118 "2022-03-08T16:41:13Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [March 8, 2022, 4:41pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118/1 "2022-03-08T16:41:13Z")

</div>

Hey I am setting up an observaiblity use case to test it with docker, and I want to collect Elasticsearch logs (gc, audit, etc.) using Filebeat.

I have Elasticsearch running in a docker container, and I have filebeat running in another container, what configuration I need to collect logs ?

From [Collecting Elasticsearch log data with Filebeat](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-filebeat.html) it says that I have to install filebeat in the same host or VM where Elasticsearch is running, but I am in a docker context should I build my own `Dockerfile` that has Elasticsearch and filebeat running in the same container? Can't find any related information from the official documentation, found some webinars but they don't cover the steps.

One more thing to add, the following lines in `filebeat.yml`:

```auto
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

```

retrieve what's printing out in the console when I run `docker-compose up`, and in Logs apm I have this:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/c/cc15fe8f14322657b1a27e8418c5d6409c19d4c4.png)

How can I replace **unknown** with a proper name like **docker-logging**?

Thanks!

---

<div class="post-metadata">

**Author:** ![Tetiana\_Kravchenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tetiana_kravchenko/32/102683_2.png) [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)\
**Post date:** [March 9, 2022, 1:59pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118/2 "2022-03-09T13:59:35Z")

</div>

Hi @marone !

> [@marone](#):
>
> From [Collecting Elasticsearch log data with Filebeat](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-filebeat.html) it says that I have to install filebeat in the same host or VM where Elasticsearch is running, but I am in a docker context should I build my own `Dockerfile` that has Elasticsearch and filebeat running in the same container?

You do not need to use custom `Dockerfile`, running filebeat as a separate container should be enough.

> I have Elasticsearch running in a docker container, and I have filebeat running in another container, what configuration I need to collect logs ?

did you check this doc - [Run Filebeat on Docker | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html) ? It seems to be quite close to what you are trying to achieve. Did you add add needed labels to the elasticsearch container to be able to use hints based autodiscover - [Hints based autodiscover | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#_docker_3) ?

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [March 9, 2022, 4:35pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118/3 "2022-03-09T16:35:58Z")

</div>

thank you @Tetiana_Kravchenko for answering back, to be honest I already saw the label with autodiscover but didn't understand how to make it working, here is what I did based on the doc:

- I added labels for Elasticsearch service in docker-compose.yml:

```auto
# ...
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    deploy:
      labels:
        co.elastic.logs/module: elasticsearch
        co.elastic.logs/fileset.stdout: access
        co.elastic.logs/fileset.stderr: error
    environment:
    - bootstrap.memory_lock=true
    - cluster.name=docker-cluster
    - cluster.routing.allocation.disk.threshold_enabled=false
    - discovery.type=single-node
    - ES_JAVA_OPTS=-XX:UseAVX=2 -Xms1g -Xmx1g
    ulimits:
      memlock:
        hard: -1
        soft: -1
    volumes:
    - esdata:/usr/share/elasticsearch/data
    ports:
    - 9200:9200
    networks:
    - elastic
    healthcheck:
      interval: 20s
      retries: 10
      test: curl -s http://localhost:9200/_cluster/health | grep -vq '"status":"red"'

  filebeat:
      container_name: filebeat
      hostname: "metricbeat"
      image: docker.elastic.co/beats/filebeat:7.15.2
      user: root
      volumes:
        - /var/lib/docker/containers:/var/lib/docker/containers:ro
        - /var/run/docker.sock:/var/run/docker.sock:ro
        - ./config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      command: ["--strict.perms=false", "-system.hostfs=/hostfs"]
      networks:
        - elastic
      depends_on:
        - elasticsearch
        - kibana
      restart: always

# ...

```

and in `filebeat.yml`:

```auto
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*-${data.container.id}.log # CRI path

processors:
- add_cloud_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

setup.dashboards.enabled: true

setup.kibana:
  host: kibana:5601

```

in APM UI I didn't get any logs 😕

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/9/b98d76c965b360db4a58d2edf0457283afa5acfc.png)

what's wrong with the conf please?

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [March 10, 2022, 3:15pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118/4 "2022-03-10T15:15:56Z")

</div>

**EDIT**

Forgot to mention that I am using Docker desktop on windows 10, and I activated logging for filebeat and have the following error: ` ERROR metrics/metrics.go:297 cgroups data collection disabled: error finding subsystems: cgroups not found or unsupported by os`

---

<div class="post-metadata">

**Author:** ![Tetiana\_Kravchenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tetiana_kravchenko/32/102683_2.png) [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)\
**Post date:** [March 14, 2022, 1:14pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118/5 "2022-03-14T13:14:48Z")

</div>

Hi @marone !

Sorry for the late reply and thank you for the detailed explanation!

Did you check if logs are actually ingested? You can check `Discover` in Kibana, you should also change the index pattern to the one where logs are ingested in (for example `filebeat-*`)

From the first look: I think you are using wrong `hints.default_config.paths` - `/var/log/containers/*-${data.container.id}.log` is mainly used for kubernetes environment, could you try `/var/lib/docker/containers/*/*.log` ?

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [March 14, 2022, 3:30pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118/6 "2022-03-14T15:30:58Z")

</div>

Thank you a lot @Tetiana_Kravchenko you saved me a lot of time, I did the changes you wrote above and here is the `filebeat.yml` for people who will see this discussion in future:

```nohighlight
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/lib/docker/containers/*/*.log # CRI path

# processors:
# - add_cloud_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

logging.level: error
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640

setup.dashboards.enabled: true

setup.kibana:
  host: kibana:5601

```

But one more last question, in [Autodiscover for docker](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#_docker_3) the `hints.default_config.paths` for docker is the same as Kubernetes 😄 it's a typo error I guess, isn't it? Confirm it to me please so I can do a pull request to fix the doc. I guess it is!

---

<div class="post-metadata">

**Author:** ![Tetiana\_Kravchenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tetiana_kravchenko/32/102683_2.png) [@Tetiana\_Kravchenko](https://discuss.elastic.co/u/Tetiana_Kravchenko)\
**Post date:** [March 14, 2022, 6:04pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118/7 "2022-03-14T18:04:33Z")

</div>

I think it is a typo, for docker should be used `/var/lib/docker/containers/*/*.log`. And if I am not mistaken - `/var/lib/docker/containers/*/*.log` is a default value, so

```auto
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

```

might work without default config defined.

---

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [March 15, 2022, 3:10pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118/8 "2022-03-15T15:10:19Z")

</div>

One more thing to mention, even with the new configuration, I have logs but they are not recognized as Elasticsearch logs, still I have 'unkown' (see image below). How can I add information, like for ex: instead of `unkown` I have `elasticsearch-gc` logs.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/7/e7276fabb5404c9bb9f03347be58c45313519b68.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 12, 2022, 3:10pm UTC](https://discuss.elastic.co/t/how-to-collect-containerized-elasticsearch-logs-with-filebeat/299118/9 "2022-04-12T15:10:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.