How to exclude other namespaces?

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: false
      hints.default_config:
        type: container
        finished: true
        paths:
          - "/var/log/containers/*.log"
      templates:
        - condition:
            or:
              - equals:
                  kubernetes.namespace: "front"
              - equals:
                  kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*.log"
setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:
  - add_kubernetes_metadata:
      in_cluster: true
      host: ${NODE_NAME}
      default_matchers.enabled: false
      matchers:
      - logs_path:
          logs_path: "/var/log/containers/"
          resource_type: "container"

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

I only need the logs under the back and front namespaces.
The above method did not succeed. How should I modify it?


In addition, I have a question, why filebeat will inject information into application events.

Hey @wajika,

In your configuration you are using a path with a wildcard that would match all the containers in the node. So every configuration generated, for every pod, will try to harvest any file. You need to setup autodiscover in a way that it generates an specific configuration for each container. Also, when using autodiscover, you don't need to use add_kubernetes_metadata, events should be already enriched by the autodiscover provider.

I think that something like this would work for you:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            or:
              - equals:
                  kubernetes.namespace: "front"
              - equals:
                  kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

Other option could be to use hints-based autodiscover, that would allow you to enable collection of logs per namespace using annotations in the namespaces themselves. See my comment about that on this topic: Collect logs for specific containers or namespace in Openshift/Kubernetes

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            or:
              - equals:
                  kubernetes.namespace: "front"
              - equals:
                  kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
              multiline.pattern: '^[[:space:]]'
              multiline.negate: false
              multiline.match: after

setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

I tried to use your configuration, but filebeat reported an error.

ERROR [autodiscover] autodiscover/autodiscover.go:209 Auto discover config check failed for config '{
"docker-json": {
"cri_flags": true,
"format": "auto",
"partial": true,
"stream": "all"
},
"multiline": {
"match": "after",
"negate": false,
"pattern": "^[[:space:]]"
},
"symlinks": true,
"type": "container"
}', won't start runner: each input must have at least one path defined

One more thing, if I use ${data.kubernetes.container.id}, can the container.id field be generated?
I need to use a field to contact other beats data (APM and metricbeat) on the kubernetes cluster

filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
add_resource_metadata:
namespace:
enabled: true

If I use this configuration, can filebeat support both pod annotations and node annotations?

I feel that using the "co.elastic.logs/enabled:'false'" method to turn off the log collection of the namespace is not optimal. If there are a lot of namespaces, then I must increase them one by one.

I think the error can be ignored in your case, but you can add a condition to ignore events without container ids, so the error doesn't happen.

Try this configuration:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            and:
              - has_fields: ['kubernetes.container.id']
              - or:
                - equals:
                    kubernetes.namespace: "front"
                - equals:
                    kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

Try with this to add node annotations:

      add_resource_metadata:
        node:
          enabled: true
          include_annotations:
            - "someannotation"
            - "someotherannotation"

Take into account that you wouldn't need to change the configuration, you would only need to add co.elastic.logs/enabled:'true' annotation to the namespaces you want to collect logs from.


I added has_ fields: [' kubernetes.container.id '] still reporting errors.

However, if you say that this error is not a big problem, let it be for the time being.

OK, that's easy.

Sorry, I didn't understand you.
I also need a field container.id, can you advice me, how to do that?

Thank you for your reply.

What do you mean? Filebeat should be adding the kubernetes.container.id field to collected logs.


Kubernetes matedata all appeared, but not yet container.id.
kubernetes.container.id Where should field be added?

Umm, is this container being stopped at this moment? Looking at the code the only case where it seems possible to have the container name but not its id is when the pod is being stopped: https://github.com/elastic/beats/blob/7fbbdca91b5cdfcb943ff7f7b7312219ae9986c0/libbeat/autodiscover/providers/kubernetes/pod.go#L339

If this is a normal running container this may be a bug. Are you missing the kubernetes.container.id in all events?

I can confirm that all containers have no container.id field.

Not even in the mapping.

Is there a solution to this problem?

This seems unexpected to me, could you try with a released version, like 7.9.0?

This missing container field also occurs on APM agent (without kubernetes metadata).

Filebeat 7.9.1 will also not be generated container.id.

Hey @wajika,

I have tried to reproduce this and there seems to be actually some problem on Beats with the container ids. I have opened an issue in Github for further investigation: https://github.com/elastic/beats/issues/20982

Thanks!