I'm following up on this topic:
The "add_kubernetes_metadata" processor works only if logs are read from
/var/lib/docker/containers/*/*.log, it doesn't work with logs from
This is caused by the way the container ID is extracted from the path in the processor.
exekias from the Elastic team says
/var/log/containers/*.log are just symlinks to
/var/lib/docker/containers/*/*.log. Of course, he's right and reading logs directly from
/var/lib/docker/containers/*/*.log enables extracting the container ID, hence enriching the logs with Kubernetes metadata.
However, there are two reasons, the processor should also work with
You may want to exclude log files from certain pods, e.g. the filebeat pod itself with the
exclude_files: ['filebeat-*.log']option. That would work only in
/var/log/containers, as only the symlinks there contain the pod name.
You may want to read only the log files of docker containers used by active Kubernetes pods, not any other docker containers running on the system. That also works only by following the symlinks in
Are there any plans on changing this before the 6.0.0 release?