I’m trying to collect logs from Kubernetes nodes using Filebeat and ONLY ship them to ELK IF the logs originate from a specific Kubernetes Namespace.
So far I’ve discovered that you can define Processors which I think accomplish this. However, no matter what I do I can not get the shipped logs to be constrained. Does this look right?
add_kubernetes_metadata enhances event with fields like pod name, namespace etc.
when you specify it like you did, you enable add_kubernetes_metadata processor for events coming from this one namespace, which means other events won't be annotated with additional metadata more here
you can specify condition, which if turns out to be true, event is dropped.
maybe you can even combine add_kubernetes_metadata with drop_event in which you will check that kubernetes.namespace is not the one you want the events for
Hi @Michal_Pristas thank you very much for your response and the linked documentation. I've reviewed it and it looks like I don't need the add_kubernetes_metadata processor to accomplish what I'm trying to do after all. In fact, even without it if I expand a log entry in Kibana I can see that the kubernetes.namespace field is already present.
However, I have already tried using a Drop Events processor which does not appear to be working unfortunately. As I am still receiving logs from all namespaces, not just the one I define with the constraint. Does my syntax look correct to you?
After a buttload of fiddling around I finally got this working by moving the drop processor to the input configuration file instead of the filebeat-config file.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.