How to exclude other namespaces?

wajika · August 27, 2020, 4:00am

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: false
      hints.default_config:
        type: container
        finished: true
        paths:
          - "/var/log/containers/*.log"
      templates:
        - condition:
            or:
              - equals:
                  kubernetes.namespace: "front"
              - equals:
                  kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*.log"
setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:
  - add_kubernetes_metadata:
      in_cluster: true
      host: ${NODE_NAME}
      default_matchers.enabled: false
      matchers:
      - logs_path:
          logs_path: "/var/log/containers/"
          resource_type: "container"

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

I only need the logs under the back and front namespaces.
The above method did not succeed. How should I modify it?

wajika · August 27, 2020, 6:51am

In addition, I have a question, why filebeat will inject information into application events.

jsoriano · August 27, 2020, 11:23am

Hey @wajika,

In your configuration you are using a path with a wildcard that would match all the containers in the node. So every configuration generated, for every pod, will try to harvest any file. You need to setup autodiscover in a way that it generates an specific configuration for each container. Also, when using autodiscover, you don't need to use add_kubernetes_metadata, events should be already enriched by the autodiscover provider.

I think that something like this would work for you:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            or:
              - equals:
                  kubernetes.namespace: "front"
              - equals:
                  kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

Other option could be to use hints-based autodiscover, that would allow you to enable collection of logs per namespace using annotations in the namespaces themselves. See my comment about that on this topic: Collect logs for specific containers or namespace in Openshift/Kubernetes

wajika · August 28, 2020, 12:31am

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            or:
              - equals:
                  kubernetes.namespace: "front"
              - equals:
                  kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
              multiline.pattern: '^[[:space:]]'
              multiline.negate: false
              multiline.match: after

setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

I tried to use your configuration, but filebeat reported an error.

ERROR [autodiscover] autodiscover/autodiscover.go:209 Auto discover config check failed for config '{
"docker-json": {
"cri_flags": true,
"format": "auto",
"partial": true,
"stream": "all"
},
"multiline": {
"match": "after",
"negate": false,
"pattern": "^[[:space:]]"
},
"symlinks": true,
"type": "container"
}', won't start runner: each input must have at least one path defined

One more thing, if I use ${data.kubernetes.container.id}, can the container.id field be generated?
I need to use a field to contact other beats data (APM and metricbeat) on the kubernetes cluster

wajika · August 28, 2020, 1:08am

filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
add_resource_metadata:
namespace:
enabled: true

If I use this configuration, can filebeat support both pod annotations and node annotations?

wajika · August 28, 2020, 5:26am

I feel that using the "co.elastic.logs/enabled:'false'" method to turn off the log collection of the namespace is not optimal. If there are a lot of namespaces, then I must increase them one by one.

jsoriano · August 30, 2020, 11:28am

I think the error can be ignored in your case, but you can add a condition to ignore events without container ids, so the error doesn't happen.

Try this configuration:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            and:
              - has_fields: ['kubernetes.container.id']
              - or:
                - equals:
                    kubernetes.namespace: "front"
                - equals:
                    kubernetes.namespace: "back"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
setup.template.enabled: false
processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOST}

jsoriano · August 30, 2020, 11:33am

Try with this to add node annotations:

      add_resource_metadata:
        node:
          enabled: true
          include_annotations:
            - "someannotation"
            - "someotherannotation"

jsoriano · August 30, 2020, 11:35am

Take into account that you wouldn't need to change the configuration, you would only need to add co.elastic.logs/enabled:'true' annotation to the namespaces you want to collect logs from.

wajika · August 31, 2020, 1:07am

I added has_ fields: [' kubernetes.container.id '] still reporting errors.

However, if you say that this error is not a big problem, let it be for the time being.

wajika · August 31, 2020, 1:09am

OK, that's easy.

wajika · August 31, 2020, 1:33am

Sorry, I didn't understand you.
I also need a field container.id, can you advice me, how to do that?

Thank you for your reply.

jsoriano · August 31, 2020, 9:00am

What do you mean? Filebeat should be adding the kubernetes.container.id field to collected logs.

wajika · August 31, 2020, 9:26am

Kubernetes matedata all appeared, but not yet container.id.
kubernetes.container.id Where should field be added?

jsoriano · August 31, 2020, 9:38am

Umm, is this container being stopped at this moment? Looking at the code the only case where it seems possible to have the container name but not its id is when the pod is being stopped: https://github.com/elastic/beats/blob/7fbbdca91b5cdfcb943ff7f7b7312219ae9986c0/libbeat/autodiscover/providers/kubernetes/pod.go#L339

If this is a normal running container this may be a bug. Are you missing the kubernetes.container.id in all events?

wajika · September 1, 2020, 1:02am

I can confirm that all containers have no container.id field.

Not even in the mapping.

wajika · September 3, 2020, 1:04am

Is there a solution to this problem?

jsoriano · September 3, 2020, 9:23am

This seems unexpected to me, could you try with a released version, like 7.9.0?

wajika · September 4, 2020, 12:52am

This missing container field also occurs on APM agent (without kubernetes metadata).

Filebeat 7.9.1 will also not be generated container.id.

jsoriano · September 4, 2020, 10:48am

Hey @wajika,

I have tried to reproduce this and there seems to be actually some problem on Beats with the container ids. I have opened an issue in Github for further investigation: https://github.com/elastic/beats/issues/20982

Thanks!