# Add\_kubernetes\_metadata should work in "/var/log/containers" **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 22, 2017, 4:54pm UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945 "2017-08-22T16:54:12Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Sven\_Woltmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sven_woltmann/32/21380_2.png) [@Sven\_Woltmann](https://discuss.elastic.co/u/Sven_Woltmann)\ **Post date:** [August 22, 2017, 4:54pm UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945/1 "2017-08-22T16:54:12Z")
I'm following up on this topic: > [@Kubernetes metadata](https://discuss.elastic.co/t/kubernetes-metadata/90865): > > I'm trying to test the new kubernetes metadata features that were added to beats and therefore would like to pull the docker image for the alpha, but it appears it is not being published. Could it be published please? Also appears the github repo links to the logstash documentation - [https://github.com/elastic/beats-docker](https://github.com/elastic/beats-docker) The "add\_kubernetes\_metadata" processor works only if logs are read from `/var/lib/docker/containers/*/*.log`, it doesn't work with logs from `/var/log/containers/*.log`. This is caused by the way the container ID is extracted from the path in the processor. exekias from the Elastic team says `/var/log/containers/*.log` are just symlinks to `/var/lib/docker/containers/*/*.log`. Of course, he's right and reading logs directly from `/var/lib/docker/containers/*/*.log` enables extracting the container ID, hence enriching the logs with Kubernetes metadata. However, there are two reasons, the processor should also work with `/var/log/containers/*.log`: 1. You may want to exclude log files from certain pods, e.g. the filebeat pod itself with the `exclude_files: ['filebeat-*.log']` option. That would work only in `/var/log/containers`, as only the symlinks there contain the pod name. 2. You may want to read only the log files of docker containers used by _active_ Kubernetes pods, not any other docker containers running on the system. That also works only by following the symlinks in `/var/log/containers`. Are there any plans on changing this before the 6.0.0 release? ---
**Author:** ![Sven\_Woltmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sven_woltmann/32/21380_2.png) [@Sven\_Woltmann](https://discuss.elastic.co/u/Sven_Woltmann)\ **Post date:** [August 23, 2017, 10:06am UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945/2 "2017-08-23T10:06:31Z")
I just found a third reason while analyzing my logs: 1. The "source" field in the log documents would be much more informative if it contained a value like `/var/log/containers/kube-proxy-4d7nt_kube-system_kube-proxy-1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266.log` instead of `/var/lib/docker/containers/1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266/1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266-json.log` ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [August 23, 2017, 11:01am UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945/3 "2017-08-23T11:01:20Z")
Hi @Sven_Woltmann, I think we can consider including support for that, could you please open a new enhancement request in githib [https://github.com/elastic/beats/issues](https://github.com/elastic/beats/issues)? I'm not sure it will make it to the 6.0 cut as it's under feature freeze already, but next version is always around the corner 😉 Thank you for your feedback! ---
**Author:** ![Sven\_Woltmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sven_woltmann/32/21380_2.png) [@Sven\_Woltmann](https://discuss.elastic.co/u/Sven_Woltmann)\ **Post date:** [August 23, 2017, 12:09pm UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945/4 "2017-08-23T12:09:25Z")
Thank you for your response. I'll try to implement the changes myself first, and will - upon success - include a pull request in the ticket. I haven't written any Go code yet - but it seems to be a nice task to get familiar with Go. ---
**Author:** ![Sven\_Woltmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sven_woltmann/32/21380_2.png) [@Sven\_Woltmann](https://discuss.elastic.co/u/Sven_Woltmann)\ **Post date:** [August 23, 2017, 6:28pm UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945/5 "2017-08-23T18:28:52Z")
Hi @exekias, I've created a pull requests here: [https://github.com/elastic/beats/pull/4981](https://github.com/elastic/beats/pull/4981) Do I also need to add an enhancement request in "Issues"? ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [August 23, 2017, 11:19pm UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945/6 "2017-08-23T23:19:27Z")
Thank you for taking the time! There is no need for a new Issue, although it's a good practice to open them, next time perhaps 🙂 ---
**Author:** ![Sven\_Woltmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sven_woltmann/32/21380_2.png) [@Sven\_Woltmann](https://discuss.elastic.co/u/Sven_Woltmann)\ **Post date:** [August 24, 2017, 10:27am UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945/7 "2017-08-24T10:27:23Z")
I’ve created a second pull requests with a cleaner and more generic solution: [https://github.com/elastic/beats/pull/4995](https://github.com/elastic/beats/pull/4995) ---
**Author:** ![Sven\_Woltmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sven_woltmann/32/21380_2.png) [@Sven\_Woltmann](https://discuss.elastic.co/u/Sven_Woltmann)\ **Post date:** [August 25, 2017, 8:31am UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945/8 "2017-08-25T08:31:44Z")
Here's a third pull request that solves the issue without requiring a processor configuration and without regular expressions (more details in the PR): [https://github.com/elastic/beats/pull/5011](https://github.com/elastic/beats/pull/5011) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 22, 2017, 8:32am UTC](https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945/9 "2017-09-22T08:32:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.