# How to constrain Filebeat to only ship logs if they contain a specific field?

**URL:** <https://discuss.elastic.co/t/how-to-constrain-filebeat-to-only-ship-logs-if-they-contain-a-specific-field/174883>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 1, 2019, 10:21pm UTC](https://discuss.elastic.co/t/how-to-constrain-filebeat-to-only-ship-logs-if-they-contain-a-specific-field/174883 "2019-04-01T22:21:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![zimmertr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zimmertr/32/43261_2.png) [@zimmertr](https://discuss.elastic.co/u/zimmertr)\
**Post date:** [April 1, 2019, 10:21pm UTC](https://discuss.elastic.co/t/how-to-constrain-filebeat-to-only-ship-logs-if-they-contain-a-specific-field/174883/1 "2019-04-01T22:21:13Z")

</div>

I’m trying to collect logs from Kubernetes nodes using Filebeat and ONLY ship them to ELK IF the logs originate from a specific Kubernetes Namespace.

So far I’ve discovered that you can define _Processors_ which I think accomplish this. However, no matter what I do I can not get the shipped logs to be constrained. Does this look right?

Hm, does this look correct then?

```
filebeat.config:
  inputs:
    path: ${path.config}/inputs.d/*.yml
    reload.enabled: true
    reload.period: 10s
    when.contains:
      kubernetes.namespace: "NAMESPACE"
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
  processors:
    - add_kubernetes_metadata:
      namespace: "NAMESPACE"
xpack.monitoring.enabled: true
output.elasticsearch:
  hosts: ['elasticsearch:9200']

```

Despite this configuration I still get logs from all of the namespaces.

Filebeat is running as a DaemonSet on Kubernetes. Here is an example of an expanded log entry: [https://i.imgur.com/xfTwbhl.png](https://i.imgur.com/xfTwbhl.png)

---

<div class="post-metadata">

**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\
**Post date:** [April 2, 2019, 11:42am UTC](https://discuss.elastic.co/t/how-to-constrain-filebeat-to-only-ship-logs-if-they-contain-a-specific-field/174883/2 "2019-04-02T11:42:39Z")

</div>

Hey @zimmertr

`add_kubernetes_metadata` enhances event with fields like pod name, namespace etc.  
when you specify it like you did, you enable `add_kubernetes_metadata` processor for events coming from this one namespace, which means other events won't be annotated with additional metadata [more here](https://www.elastic.co/guide/en/beats/filebeat/6.7/add-kubernetes-metadata.html)

what you probably need is [Drop events processor](https://www.elastic.co/guide/en/beats/filebeat/6.7/drop-event.html)

you can specify condition, which if turns out to be true, event is dropped.  
maybe you can even combine `add_kubernetes_metadata` with `drop_event` in which you will check that `kubernetes.namespace` is not the one you want the events for

---

<div class="post-metadata">

**Author:** ![zimmertr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zimmertr/32/43261_2.png) [@zimmertr](https://discuss.elastic.co/u/zimmertr)\
**Post date:** [April 2, 2019, 6:26pm UTC](https://discuss.elastic.co/t/how-to-constrain-filebeat-to-only-ship-logs-if-they-contain-a-specific-field/174883/3 "2019-04-02T18:26:19Z")

</div>

Hi @Michal_Pristas thank you very much for your response and the linked documentation. I've reviewed it and it looks like I don't need the `add_kubernetes_metadata` processor to accomplish what I'm trying to do after all. In fact, even without it if I expand a log entry in Kibana I can see that the `kubernetes.namespace` field is already present.

However, I have already tried using a Drop Events processor which does not appear to be working unfortunately. As I am still receiving logs from all namespaces, not just the one I define with the constraint. Does my syntax look correct to you?

```
filebeat.config:
  inputs:
    path: ${path.config}/inputs.d/*.yml
    reload.enabled: true
    reload.period: 10s
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
  processors:
  - drop_event:
      when:
        not:
          equals:
            kubernetes.namespace: NAMESPACE
xpack.monitoring.enabled: true
output.elasticsearch:
    hosts: ['elasticsearch:9200']
```

---

<div class="post-metadata">

**Author:** ![zimmertr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zimmertr/32/43261_2.png) [@zimmertr](https://discuss.elastic.co/u/zimmertr)\
**Post date:** [April 2, 2019, 11:41pm UTC](https://discuss.elastic.co/t/how-to-constrain-filebeat-to-only-ship-logs-if-they-contain-a-specific-field/174883/4 "2019-04-02T23:41:37Z")

</div>

After a buttload of fiddling around I finally got this working by moving the drop processor to the input configuration file instead of the filebeat-config file.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 30, 2019, 11:41pm UTC](https://discuss.elastic.co/t/how-to-constrain-filebeat-to-only-ship-logs-if-they-contain-a-specific-field/174883/5 "2019-04-30T23:41:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.