How to constrain Filebeat to only ship logs if they contain a specific field?

[zimmertr (TJ Zimmerman)]

I’m trying to collect logs from Kubernetes nodes using Filebeat and ONLY ship them to ELK IF the logs originate from a specific Kubernetes Namespace.

So far I’ve discovered that you can define Processors which I think accomplish this. However, no matter what I do I can not get the shipped logs to be constrained. Does this look right?

Hm, does this look correct then?

filebeat.config:
  inputs:
    path: ${path.config}/inputs.d/*.yml
    reload.enabled: true
    reload.period: 10s
    when.contains:
      kubernetes.namespace: "NAMESPACE"
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
  processors:
    - add_kubernetes_metadata:
      namespace: "NAMESPACE"
xpack.monitoring.enabled: true
output.elasticsearch:
  hosts: ['elasticsearch:9200']

Despite this configuration I still get logs from all of the namespaces.

Filebeat is running as a DaemonSet on Kubernetes. Here is an example of an expanded log entry: https://i.imgur.com/xfTwbhl.png

[Michal_Pristas]

Hey @zimmertr

add_kubernetes_metadata enhances event with fields like pod name, namespace etc.
when you specify it like you did, you enable add_kubernetes_metadata processor for events coming from this one namespace, which means other events won't be annotated with additional metadata more here

what you probably need is Drop events processor

you can specify condition, which if turns out to be true, event is dropped.
maybe you can even combine add_kubernetes_metadata with drop_event in which you will check that kubernetes.namespace is not the one you want the events for

[zimmertr (TJ Zimmerman)]

Hi @Michal_Pristas thank you very much for your response and the linked documentation. I've reviewed it and it looks like I don't need the add_kubernetes_metadata processor to accomplish what I'm trying to do after all. In fact, even without it if I expand a log entry in Kibana I can see that the kubernetes.namespace field is already present.

However, I have already tried using a Drop Events processor which does not appear to be working unfortunately. As I am still receiving logs from all namespaces, not just the one I define with the constraint. Does my syntax look correct to you?

filebeat.config:
  inputs:
    path: ${path.config}/inputs.d/*.yml
    reload.enabled: true
    reload.period: 10s
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
  processors:
  - drop_event:
      when:
        not:
          equals:
            kubernetes.namespace: NAMESPACE
xpack.monitoring.enabled: true
output.elasticsearch:
    hosts: ['elasticsearch:9200']

[zimmertr (TJ Zimmerman)]

After a buttload of fiddling around I finally got this working by moving the drop processor to the input configuration file instead of the filebeat-config file.