Collect logs for specific containers or namespace in Openshift/Kubernetes

[sayeedc]

Can filebeat daemonsets be configured to:

1. Collect logs for specific pods in a namespace and nothing else.
2. Collect logs for all pods in specific namespaces and nothing else.

I want to avoid filebeat picking up all container logs from every node. The other option is to run it as sidecar with the relevant pod but I'd like to avoid this approach.

[jsoriano (Jaime Soriano)]

Hey @sayeedc,

Yes, you can use hints-based autodiscover to provide a different configuration per pod or namespace. Hints are defined as annotations in your kubernetes resources.

In the main configuration file you would need this setup:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      add_resource_metadata.namespace.enabled: true
      hints.default_config.enabled: false

This tells the autodiscover provider to enable hints, to take into account namespace metadata, and as default configuration for each pod, it disables the inputs.

Then you can add configurations per namespace, or per pod. Adding an annotation like the following one will enable the inputs selectively:

annotations:
  co.elastic.logs/enabled: 'true'

If you do it in a pod, its logs will be collected. If you do it in a namespace, the logs of all the pods in this namespace will be collected.

You can see in the documentation other settings you can use per pod or namespace, apart of co.elastic.logs/enabled: https://www.elastic.co/guide/en/beats/filebeat/7.7/configuration-autodiscover-hints.html