Add_kubernetes_metadata should work in "/var/log/containers"

[Sven_Woltmann]

I'm following up on this topic:

Kubernetes metadata

I'm trying to test the new kubernetes metadata features that were added to beats and therefore would like to pull the docker image for the alpha, but it appears it is not being published. Could it be published please? Also appears the github repo links to the logstash documentation - https://github.com/elastic/beats-docker

The "add_kubernetes_metadata" processor works only if logs are read from /var/lib/docker/containers/*/*.log, it doesn't work with logs from /var/log/containers/*.log.

This is caused by the way the container ID is extracted from the path in the processor.

exekias from the Elastic team says /var/log/containers/*.log are just symlinks to /var/lib/docker/containers/*/*.log. Of course, he's right and reading logs directly from /var/lib/docker/containers/*/*.log enables extracting the container ID, hence enriching the logs with Kubernetes metadata.

However, there are two reasons, the processor should also work with /var/log/containers/*.log:

1. You may want to exclude log files from certain pods, e.g. the filebeat pod itself with the exclude_files: ['filebeat-*.log'] option. That would work only in /var/log/containers, as only the symlinks there contain the pod name.

2. You may want to read only the log files of docker containers used by active Kubernetes pods, not any other docker containers running on the system. That also works only by following the symlinks in /var/log/containers.

Are there any plans on changing this before the 6.0.0 release?

[Sven_Woltmann]

I just found a third reason while analyzing my logs:

3. The "source" field in the log documents would be much more informative if it contained a value like /var/log/containers/kube-proxy-4d7nt_kube-system_kube-proxy-1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266.log
   instead of
   /var/lib/docker/containers/1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266/1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266-json.log

[exekias (Carlos Pérez Aradros)]

Hi @Sven_Woltmann,

I think we can consider including support for that, could you please open a new enhancement request in githib https://github.com/elastic/beats/issues?

I'm not sure it will make it to the 6.0 cut as it's under feature freeze already, but next version is always around the corner

Thank you for your feedback!

[Sven_Woltmann]

Thank you for your response. I'll try to implement the changes myself first, and will - upon success - include a pull request in the ticket.

I haven't written any Go code yet - but it seems to be a nice task to get familiar with Go.

[Sven_Woltmann]

Hi @exekias,

I've created a pull requests here: https://github.com/elastic/beats/pull/4981

Do I also need to add an enhancement request in "Issues"?

[exekias (Carlos Pérez Aradros)]

Thank you for taking the time!

There is no need for a new Issue, although it's a good practice to open them, next time perhaps

[Sven_Woltmann]

I’ve created a second pull requests with a cleaner and more generic solution: https://github.com/elastic/beats/pull/4995

[Sven_Woltmann]

Here's a third pull request that solves the issue without requiring a processor configuration and without regular expressions (more details in the PR): https://github.com/elastic/beats/pull/5011