Filebeat for kubernetes containers. How include / exclude logs by kubernetes labes?

Elastic Stack Beats
1

Hi,

I have a kubernetes cluster (currently in dev status) where following is running:

  • elastic stack
  • different project specific apps.

I want to use the same elastic stack for monitoring my kubernetes instance and my project apps. (Yes, I am aware of the fact, if my kubernetes goes down kompletely, my elastic stack will also fall down).

I do not want to ship logs of the elastic stack to elasticsearch, because this insert may lead to more logs in elasticsearch, so I could end up in some infinity loop. :frowning:

In your sample for running filebeat in kubernetes at https://www.elastic.co/de/blog/shipping-kubernetes-logs-to-elasticsearch-with-filebeat I saw, that the file path of the prospector is simply set to - /var/lib/docker/containers/*/*.log.

This would grap all container logs!

What is the most easy and resource efficient way to define dynamically which logs should be shipped to logstash and which not?

  1. I could send everything to logstash and drop it there by checking the metadata. For example I could introduce a label saveLogsInEs=true. If this is set, process the log, otherwise drop it. But I assume this is quite a resource overkill.

  2. Or is there a way to drop in filebeat to drop by kubernetes label before sending to logstash? In this case the log lines are read, enriched, then dropped. I would like to avoid it if possible.

  3. Or at best, is there a possibility to exclude prospector paths by kubernetes labels? I think this would be the optimal way to reduce unneeded load, because the file does not even need to be tailed.

Thanks a lot,
Andreas

2

You can probably get what you want by using the exclude_paths option in filebeat to prevent ingestion of the elastic stack logs, see the log input configuration settings.

3

@faec, thanks for your reply.

But in exclude_paths I need to know at path level, which containers I need and wich not.
Docker is using uids for naming container pathes. So I have no connection between container path on disk and label or deployment name.
In a highly dynamic deployment where a container may have short lifetime it is not possible to blacklist in filebeat based on container path.

2 Likes
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.