Filebeat Inputs: Docker vs Log with options

Elastic Stack Beats
1

Out of curiosity, what's the difference between the two:

- type: log
  paths:
    - /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: log
## assume downstream logstash adds stream & timestamp handling on fields

- type: docker
  containers:
    path: "/var/lib/docker/containers"
    stream: "all"
    ids:
      - "*"

Is there additional docker data added (not referring to the add_docker_metadata processor, purely input functionality only).

Thanks!
Justin

2

Hi @justinw,

Actually with these configurations the results would be pretty similar for lots of cases, and indeed docker and log inputs share some code, but at the end they are quite different.

Log input is a generic input for files and in a basic and generic way it is able to collect any log file, usually also parsing its content, as is the case for JSON files, so with a proper configuration it could be able to do something with docker logs too. It has also many options that are not needed on docker logs.

On the other hand docker input is very specific to docker, it parses JSON log lines, yes, but it also knows what to expect in the parsed objects (like the kind of stream or the timestamp), where to find the logs for an specific container and things like that.
Docker input is also able to parse logs in CRI log format, that is not JSON. This cannot be done with a single log input configuration, and the logic needed to handle these kind of logs is not included in any other input.

Having an specific input for docker comes pretty handy, it works by default for this common use case, and the exposed options are very specific. It can also handle particularities of different versions or log formats. Doing it from a generic log input would over-complicate it, and would require complex and error prone configurations.

I hope this satisfies your curiosity :slight_smile:

3

Makes sense. Good read on the design proposal. Thanks for including that.

Curiosity satisfied, thanks @jsoriano!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.