Add_kubernetes_metadata should work in "/var/log/containers"

Elastic Stack Beats
1

I'm following up on this topic:

The "add_kubernetes_metadata" processor works only if logs are read from /var/lib/docker/containers/*/*.log, it doesn't work with logs from /var/log/containers/*.log.

This is caused by the way the container ID is extracted from the path in the processor.

exekias from the Elastic team says /var/log/containers/*.log are just symlinks to /var/lib/docker/containers/*/*.log. Of course, he's right and reading logs directly from /var/lib/docker/containers/*/*.log enables extracting the container ID, hence enriching the logs with Kubernetes metadata.

However, there are two reasons, the processor should also work with /var/log/containers/*.log:

  1. You may want to exclude log files from certain pods, e.g. the filebeat pod itself with the exclude_files: ['filebeat-*.log'] option. That would work only in /var/log/containers, as only the symlinks there contain the pod name.

  2. You may want to read only the log files of docker containers used by active Kubernetes pods, not any other docker containers running on the system. That also works only by following the symlinks in /var/log/containers.

Are there any plans on changing this before the 6.0.0 release?

1 Like
2

I just found a third reason while analyzing my logs:

  1. The "source" field in the log documents would be much more informative if it contained a value like /var/log/containers/kube-proxy-4d7nt_kube-system_kube-proxy-1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266.log
    instead of
    /var/lib/docker/containers/1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266/1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266-json.log
3

Hi @Sven_Woltmann,

I think we can consider including support for that, could you please open a new enhancement request in githib https://github.com/elastic/beats/issues?

I'm not sure it will make it to the 6.0 cut as it's under feature freeze already, but next version is always around the corner :wink:

Thank you for your feedback!

4

Thank you for your response. I'll try to implement the changes myself first, and will - upon success - include a pull request in the ticket.

I haven't written any Go code yet - but it seems to be a nice task to get familiar with Go.

5

Hi @exekias,

I've created a pull requests here: https://github.com/elastic/beats/pull/4981

Do I also need to add an enhancement request in "Issues"?

1 Like
6

Thank you for taking the time!

There is no need for a new Issue, although it's a good practice to open them, next time perhaps :slight_smile:

7

I’ve created a second pull requests with a cleaner and more generic solution: https://github.com/elastic/beats/pull/4995

8

Here's a third pull request that solves the issue without requiring a processor configuration and without regular expressions (more details in the PR): https://github.com/elastic/beats/pull/5011

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.