How to exclude other namespaces?

Elastic Stack Beats
,
3

Hey @wajika,

In your configuration you are using a path with a wildcard that would match all the containers in the node. So every configuration generated, for every pod, will try to harvest any file. You need to setup autodiscover in a way that it generates an specific configuration for each container. Also, when using autodiscover, you don't need to use add_kubernetes_metadata, events should be already enriched by the autodiscover provider.

I think that something like this would work for you:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            or:
              - equals:
                  kubernetes.namespace: "front"
              - equals:
                  kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

Other option could be to use hints-based autodiscover, that would allow you to enable collection of logs per namespace using annotations in the namespaces themselves. See my comment about that on this topic: Collect logs for specific containers or namespace in Openshift/Kubernetes