Collecting logs from docker hosts

[exekias (Carlos Pérez Aradros)]

@jbacic

If I understood it well, it should work like this:

if application starts logging before filebeat is available, filebeat will tail the log and catch up with everything it missed
if logstash is not available, filebeat will wait
"buffer" size is defined by max-size and max-file logging driver options
applications log asynchronously (that is they are not blocked by logging infrastructure)
in case of performance problems, logstash can be scaled by adding additional instances (and LB between them)

Questions:

Any suggestions to the description above? Is there a better way to handle that? I've already considered gelf driver (UDP packets - logs can be lost), syslog (can be blocking for apps if not available + more parsing), logspout (it was loosing logs from the time it was down). This one seems to be the best for now.
I wrote condition based on image but finally I'd like to use labels. However, I was not able to make it work with filebeat-6.2.2. Does it mean Fix conditions checking on autodiscover Docker labels by exekias · Pull Request #6412 · elastic/beats · GitHub is not yet included?

Hi @jbacic,

Your understanding of how all this works looks good to me . Docker will write to files, and that's your buffer as of today. Filebeat is able to pick logs from there and keep a registry of reading offsets, so it can recover from restarts.

You may want to have a look to Logstash persisten quques to buffer on disk in the logstash node: Persistent queues (PQ) | Logstash Reference [8.11] | Elastic.

1. The overall architecture looks good to me, would love to hear about your experience if you go live with it

2. The fix didn't go out yet, you will have to wait for 6.2.3 or build your own image. Also take into account that dotted labels are an issue, we are working on a new fix for those.

Best regards