Creating an index for each Docker container log using Filebeats

[Joseph_Gange]

I'm using Filebeats to capture and send my Docker container logs to Elasticsearch. I am adding the Docker metadata and I would like to name each index with the associated service name. Based on the default template, I was looking at the docker.container.labels.com.docker.swarm.service.name field for my index name. I tried setting this in the filebeat.yml file but although the container started successfully, no indices were created in elasticsearch.

The relevant port of the filebeats.yml file look like this-

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

filebeat.modules:
- module: logstash

filebeat.inputs:
- type: docker
  combine_partial: true
  containers:
path: "/var/lib/docker/containers"
stream: "all"
ids:
  - "*"

processors:
- add_docker_metadata: ~

output.elasticsearch:
  hosts: ["http://elasticsearch_log:9200"]
  index: "%{[docker.container.labels.com.docker.swarm.service.name]}-%{[beat.version]}-%{+yyyy.MM.dd}"

setup.template:
  name:  "%{[docker.container.labels.com.docker.swarm.service.name]}"
  pattern: "%{[docker.container.labels.com.docker.swarm.service.name]}-*"

setup.kibana:
  host: ["kibana:5601"]

I recognize this will result in a lot indices but I am not concerned with that right now, just seeing if this is a workable configuration.

Please advise.

Thanks.

[Christian_Dahlqvist]

Having lots of small indices and shards is very inefficient and likely to cause you problems down the line. I would therefore recommend against doing this.

[Joseph_Gange]

Understood-- this is a requirement for a project. I need to demonstrate this a viable process.

Thanks!

[Christian_Dahlqvist]

I am doubt this is a viable approach. I would expect you to run into performance and stability problems quite quickly unless you only have a trivial number of containers.

It would be interesting to know why this would be a requirement.

[Joseph_Gange]

Hi Christian,
It is a trivial number of containers (about 10). And I fully understand the performance implications. Could you let me know how the configuration would work? That way I can at least POC it and then make the recommendations.

Thanks,
Joe