Best practice to ship logs from docker containers

Hi there,

Shortly I changed some applications to docker containers. Also running some components of elastic stack in docker now (logstash, kibana).

So in general docker containers are logging to stdout, which I can see with docker logs.

But what is the best practice to ship container logs with filebeat?

I am using version 6.2.3 of elasitic stack including filebeat.
Thanks in advance, Andreas

Hi @asp,

Containers stdout/stderr is stored by Docker under /var/lib/docker/containers. We have a dockerprospector that's able to read logs from there, parsing the Docker format. Also we provide with a processor that enriches logs with useful metadata from the source container, you can use a conf like this:

filebeat.prospectors:
   # Read all docker containers:
  - type: docker
    containers.ids:
      - '*'

# Enrich docker events with metadata:
processors:
  - add_docker_metadata: ~

It will need to run with enough permissions to read /var/lib/docker/containers/* and talk to /var/run/docker.sock (for enriching).

You can run filebeat as a container too, just make sure you mount /var/lib/docker/containers and /var/run/docker.sock.

Best regards

@exekias thanks a lot. I will give it a try

is there a feature to filter for container names instead of ids?

Yes,

You can use autodiscover for that:

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html

Best regards

Thanks you solved one of my questions, very kind of you.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.