Best practice to ship logs from docker containers

Elastic Stack Beats
1

Hi there,

Shortly I changed some applications to docker containers. Also running some components of elastic stack in docker now (logstash, kibana).

So in general docker containers are logging to stdout, which I can see with docker logs.

But what is the best practice to ship container logs with filebeat?

I am using version 6.2.3 of elasitic stack including filebeat.
Thanks in advance, Andreas

2

Hi @asp,

Containers stdout/stderr is stored by Docker under /var/lib/docker/containers. We have a dockerprospector that's able to read logs from there, parsing the Docker format. Also we provide with a processor that enriches logs with useful metadata from the source container, you can use a conf like this:

filebeat.prospectors:
   # Read all docker containers:
  - type: docker
    containers.ids:
      - '*'

# Enrich docker events with metadata:
processors:
  - add_docker_metadata: ~

It will need to run with enough permissions to read /var/lib/docker/containers/* and talk to /var/run/docker.sock (for enriching).

You can run filebeat as a container too, just make sure you mount /var/lib/docker/containers and /var/run/docker.sock.

Best regards

3

@exekias thanks a lot. I will give it a try

4

is there a feature to filter for container names instead of ids?

5

Yes,

You can use autodiscover for that:

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html

Best regards

6

Thanks you solved one of my questions, very kind of you.