# Filebeat autodiscover filter by namespace or container labels

**URL:** <https://discuss.elastic.co/t/filebeat-autodiscover-filter-by-namespace-or-container-labels/362550>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [July 4, 2024, 12:50pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-filter-by-namespace-or-container-labels/362550 "2024-07-04T12:50:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrea\_Bertanzon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_bertanzon/32/135800_2.png) [@Andrea\_Bertanzon](https://discuss.elastic.co/u/Andrea_Bertanzon)\
**Post date:** [July 4, 2024, 12:50pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-filter-by-namespace-or-container-labels/362550/1 "2024-07-04T12:50:36Z")

</div>

Hello,  
I deployed Filebeats in my kubernetes cluster. The cluster uses CRI-O not Docker or Containerd (this is helpful for log folder path).

The idea I am trying to implement is to make so that Filebeats only collects logs from containers that have a specific label attached to them, so that I can reduce the logs to what I really need.

To do so, I tried implementing a simpler scenarios, collecting logs only from a specific namespace: dds

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: dds
              config:
                - type: container
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  paths:
                    - /var/log/pods/*/*/*.log
    output.logstash:
      hosts: mylogstashhost

```

With this, I get tons of logs inside Kibana, also logs from pods that are not inside the dds namespace. What am I doing wrong?

How would I filter logs that are not coming from containers marked with label like logger: fb?

I thank you in advance for your assistance

---

<div class="post-metadata">

**Author:** ![Andrea\_Bertanzon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_bertanzon/32/135800_2.png) [@Andrea\_Bertanzon](https://discuss.elastic.co/u/Andrea_Bertanzon)\
**Post date:** [July 5, 2024, 8:09am UTC](https://discuss.elastic.co/t/filebeat-autodiscover-filter-by-namespace-or-container-labels/362550/2 "2024-07-05T08:09:41Z")

</div>

I tried also to use hints.... but nothing changed.

I set:

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints:
            enabled: true
            default_config:
              enabled: false
              type: container
              paths:
                - /var/log/containers/*${data.kubernetes.container.id}.log
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: dds
              config:
                - type: container
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  paths:
                    - /var/log/pods/*/*/*.log

```

according to the documentation, with hints enabled, filebeats by defaults collects logs from all the pods. If you disable the default config, you should instead annotate pods that you want to log:

[Hints based autodiscover | Filebeat Reference [8.14] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/8.14/configuration-autodiscover-hints.html) says:

> Filebeat gets logs from all containers by default, you can set this hint to `false` to ignore the output of the container. Filebeat won’t read or send logs from it. **If default config is disabled, you can use this annotation to enable log retrieval only for containers with this set to `true`**. If you are aiming to use this with Kubernetes, have in mind that annotation values can only be of string type so you will need to explicitly define this as `"true"` or `"false"` accordingly.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 6, 2024, 4:01pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-filter-by-namespace-or-container-labels/362550/3 "2024-07-06T16:01:44Z")

</div>

Hi @Andrea_Bertanzon Welcome to the community.

Autodiscover and Hints lots going on there .... and often a little confusion.

So, first, just to answer your question on filtering logs...

Here is my working version.  
Note Not using hints ... I will talk about them a bit below...

All 3 different versions of `equals` work and only ship those logs.  
I think you may have an issue with including your `path` as you filter above in the condition then that `path` says get everything from everywhere... I suspect that is your issue

I think that You need that container id in that path somewhere, pretty sure that is your issue. I am not familiar with CRI-O but I suspect you just need to build / parameterize the path properly.

```auto
                  paths:
                    - /var/log/pods/*/*/*.log

```

Working... make sure you prefix with the [right prefix if you are using labels or annotations](https://www.elastic.co/guide/en/beats/filebeat/8.14/configuration-autodiscover.html#_kubernetes)

```auto
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: kube-system
                  # kubernetes.annotations.kubectl.kubernetes.io/default-container: frontend
                  # kubernetes.labels.app.kubernetes.io/name: my-otel-demo-frontend
              config:
                - type: container
                  paths:
                    - "/var/log/containers/*-${data.kubernetes.container.id}.log"

```

Hints: The purpose of [hints](https://www.elastic.co/guide/en/beats/filebeat/8.14/configuration-autodiscover-hints.html) is to declaratively provide hints / direction to filebeat to perform additional processing (multiline, processors etc). You do this by turning on Hints and providing the proper annotations. There is a limited set of hints that are valid. Hints can be used to filter using the enabled hint if you like... But IMHO not the main purpose of hints.. it's note about the processing

In your hints example I still thing you have the same probably as above you apply conditions then you still have the paths saying collect everything...

```auto
                  paths:
                    - /var/log/pods/*/*/*.log

```

Also I am unclear where you got this directive from

```auto
                  containers.ids:

```

especially with plural `containers` perhaps I am missing something

Hope that helps some...