# Filebeat for kubernetes containers. How include / exclude logs by kubernetes labes?

**URL:** <https://discuss.elastic.co/t/filebeat-for-kubernetes-containers-how-include-exclude-logs-by-kubernetes-labes/195946>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 20, 2019, 1:36pm UTC](https://discuss.elastic.co/t/filebeat-for-kubernetes-containers-how-include-exclude-logs-by-kubernetes-labes/195946 "2019-08-20T13:36:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [August 20, 2019, 1:36pm UTC](https://discuss.elastic.co/t/filebeat-for-kubernetes-containers-how-include-exclude-logs-by-kubernetes-labes/195946/1 "2019-08-20T13:36:43Z")

</div>

Hi,

I have a kubernetes cluster (currently in dev status) where following is running:

- elastic stack
- different project specific apps.

I want to use the same elastic stack for monitoring my kubernetes instance and my project apps. (Yes, I am aware of the fact, if my kubernetes goes down kompletely, my elastic stack will also fall down).

I do not want to ship logs of the elastic stack to elasticsearch, because this insert may lead to more logs in elasticsearch, so I could end up in some infinity loop. ☹

In your sample for running filebeat in kubernetes at [https://www.elastic.co/de/blog/shipping-kubernetes-logs-to-elasticsearch-with-filebeat](https://www.elastic.co/de/blog/shipping-kubernetes-logs-to-elasticsearch-with-filebeat) I saw, that the file path of the prospector is simply set to `- /var/lib/docker/containers/*/*.log`.

This would grap **all container logs**!

What is the most easy and resource efficient way to define dynamically which logs should be shipped to logstash and which not?

1. I could send everything to logstash and drop it there by checking the metadata. For example I could introduce a label saveLogsInEs=true. If this is set, process the log, otherwise drop it. But I assume this is quite a resource overkill.

2. Or is there a way to drop in filebeat to drop by kubernetes label before sending to logstash? In this case the log lines are read, enriched, then dropped. I would like to avoid it if possible.

3. Or at best, is there a possibility to exclude prospector paths by kubernetes labels? I think this would be the optimal way to reduce unneeded load, because the file does not even need to be tailed.

Thanks a lot,  
Andreas

---

<div class="post-metadata">

**Author:** ![faec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/faec/32/46988_2.png) [@faec](https://discuss.elastic.co/u/faec)\
**Post date:** [August 22, 2019, 8:39pm UTC](https://discuss.elastic.co/t/filebeat-for-kubernetes-containers-how-include-exclude-logs-by-kubernetes-labes/195946/2 "2019-08-22T20:39:40Z")

</div>

You can probably get what you want by using the `exclude_paths` option in filebeat to prevent ingestion of the elastic stack logs, see the [log input configuration settings](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html).

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [August 23, 2019, 6:22am UTC](https://discuss.elastic.co/t/filebeat-for-kubernetes-containers-how-include-exclude-logs-by-kubernetes-labes/195946/3 "2019-08-23T06:22:43Z")

</div>

@faec, thanks for your reply.

But in exclude\_paths I need to know at path level, which containers I need and wich not.  
Docker is using uids for naming container pathes. So I have no connection between container path on disk and label or deployment name.  
In a highly dynamic deployment where a container may have short lifetime it is not possible to blacklist in filebeat based on container path.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 20, 2019, 6:22am UTC](https://discuss.elastic.co/t/filebeat-for-kubernetes-containers-how-include-exclude-logs-by-kubernetes-labes/195946/4 "2019-09-20T06:22:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.