Creating an index for each Docker container log using Filebeats

I'm using Filebeats to capture and send my Docker container logs to Elasticsearch. I am adding the Docker metadata and I would like to name each index with the associated service name. Based on the default template, I was looking at the field for my index name. I tried setting this in the filebeat.yml file but although the container started successfully, no indices were created in elasticsearch.

The relevant port of the filebeats.yml file look like this-

  path: ${path.config}/modules.d/*.yml

- module: logstash

- type: docker
  combine_partial: true
path: "/var/lib/docker/containers"
stream: "all"
  - "*"

- add_docker_metadata: ~

  hosts: ["http://elasticsearch_log:9200"]
  index: "%{[]}-%{[beat.version]}-%{+yyyy.MM.dd}"

  name:  "%{[]}"
  pattern: "%{[]}-*"

  host: ["kibana:5601"]

I recognize this will result in a lot indices but I am not concerned with that right now, just seeing if this is a workable configuration.

Please advise.


Having lots of small indices and shards is very inefficient and likely to cause you problems down the line. I would therefore recommend against doing this.

Understood-- this is a requirement for a project. I need to demonstrate this a viable process.


I am doubt this is a viable approach. I would expect you to run into performance and stability problems quite quickly unless you only have a trivial number of containers.

It would be interesting to know why this would be a requirement.

Hi Christian,
It is a trivial number of containers (about 10). And I fully understand the performance implications. Could you let me know how the configuration would work? That way I can at least POC it and then make the recommendations.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.