Filebeat autodiscover filter by namespace or container labels

Elastic Stack Beats
,
1

Hello,
I deployed Filebeats in my kubernetes cluster. The cluster uses CRI-O not Docker or Containerd (this is helpful for log folder path).

The idea I am trying to implement is to make so that Filebeats only collects logs from containers that have a specific label attached to them, so that I can reduce the logs to what I really need.

To do so, I tried implementing a simpler scenarios, collecting logs only from a specific namespace: dds

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: dds
              config:
                - type: container
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  paths:
                    - /var/log/pods/*/*/*.log
    output.logstash:
      hosts: mylogstashhost

With this, I get tons of logs inside Kibana, also logs from pods that are not inside the dds namespace. What am I doing wrong?

How would I filter logs that are not coming from containers marked with label like logger: fb?

I thank you in advance for your assistance

2

I tried also to use hints.... but nothing changed.

I set:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints:
            enabled: true
            default_config:
              enabled: false
              type: container
              paths:
                - /var/log/containers/*${data.kubernetes.container.id}.log
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: dds
              config:
                - type: container
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  paths:
                    - /var/log/pods/*/*/*.log

according to the documentation, with hints enabled, filebeats by defaults collects logs from all the pods. If you disable the default config, you should instead annotate pods that you want to log:

Hints based autodiscover | Filebeat Reference [8.14] | Elastic says:

Filebeat gets logs from all containers by default, you can set this hint to false to ignore the output of the container. Filebeat won’t read or send logs from it. If default config is disabled, you can use this annotation to enable log retrieval only for containers with this set to true . If you are aiming to use this with Kubernetes, have in mind that annotation values can only be of string type so you will need to explicitly define this as "true" or "false" accordingly.

3

Hi @Andrea_Bertanzon Welcome to the community.

Autodiscover and Hints lots going on there .... and often a little confusion.

So, first, just to answer your question on filtering logs...

Here is my working version.
Note Not using hints ... I will talk about them a bit below...

All 3 different versions of equals work and only ship those logs.
I think you may have an issue with including your path as you filter above in the condition then that path says get everything from everywhere... I suspect that is your issue

I think that You need that container id in that path somewhere, pretty sure that is your issue. I am not familiar with CRI-O but I suspect you just need to build / parameterize the path properly.

                  paths:
                    - /var/log/pods/*/*/*.log

Working... make sure you prefix with the right prefix if you are using labels or annotations

data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: kube-system
                  # kubernetes.annotations.kubectl.kubernetes.io/default-container: frontend
                  # kubernetes.labels.app.kubernetes.io/name: my-otel-demo-frontend
              config:
                - type: container
                  paths:
                    - "/var/log/containers/*-${data.kubernetes.container.id}.log"

Hints: The purpose of hints is to declaratively provide hints / direction to filebeat to perform additional processing (multiline, processors etc). You do this by turning on Hints and providing the proper annotations. There is a limited set of hints that are valid. Hints can be used to filter using the enabled hint if you like... But IMHO not the main purpose of hints.. it's note about the processing

In your hints example I still thing you have the same probably as above you apply conditions then you still have the paths saying collect everything...

                  paths:
                    - /var/log/pods/*/*/*.log

Also I am unclear where you got this directive from

                  containers.ids:

especially with plural containers perhaps I am missing something

Hope that helps some...