Filebeat kubernetes autodiscovery per namespace & kibana missing beats

Elastic Stack Beats
1

I've used this as a starting point to get ECK up and running:
github/elastic/cloud-on-k8s/2.7/config/recipes/beats/stack_monitoring.yaml

I just want metrics and logs collected for elastic in one namespace, and the filebeat example provided sends all logs to my monitoring cluster, into a .ds-filebeat-8.7.0 index. This grows rapidly with other installations that are running in this cluster.

So, here's the working example to start with:

  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          scope: node
          hints:
            enabled: true
            default_config:
              type: container
              paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

I tried filtering on the namespace with a condition in the template, like so:

  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.default_config.enabled: "false"
          hints.enabled: "true"
          templates:
          - condition.equals.kubernetes.namespace: elastic
            config:
            - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
              type: container

I see in the filebeat logs that it is detecting the logs for the correct containers with the elastic namespace. I see that the .ds-filebeat index grows, but Kibana does not show these logs and it tells me that Beats were not detected.

Next, I tried adding the co.elastic.logs/enabled: "true" annotation to all my elastic and kibana pods. ECK appears to add another co.elastic.logs/module: elasticsearch annotation to the running pods and then I use this for the autodiscovery configuration:

  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          hints.default_config.enabled: "false"
          hints.enabled: "true"

I obtained this from this documentation section: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#_kubernetes_2
Again, I see that in filebeat's logs, that it is detecting the pods with this annotation, but Kibana again does not pick up that these are elastic logs.

How do I configure filebeat autosdiscover to collect from one namespace and have Kibana pick this up? Is there a way to debug why Kibana is not picking up the logs in the "Stack Monitoring" view?

2

Since I can't add github links(?), here's a link to the link:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.