# Filebeat kubernetes autodiscovery per namespace & kibana missing beats

**URL:** <https://discuss.elastic.co/t/filebeat-kubernetes-autodiscovery-per-namespace-kibana-missing-beats/333722>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 18, 2023, 6:24am UTC](https://discuss.elastic.co/t/filebeat-kubernetes-autodiscovery-per-namespace-kibana-missing-beats/333722 "2023-05-18T06:24:05Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![bdols](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bdols/32/121151_2.png) [@bdols](https://discuss.elastic.co/u/bdols)\
**Post date:** [May 18, 2023, 6:24am UTC](https://discuss.elastic.co/t/filebeat-kubernetes-autodiscovery-per-namespace-kibana-missing-beats/333722/1 "2023-05-18T06:24:05Z")

</div>

I've used this as a starting point to get ECK up and running:  
github/elastic/cloud-on-k8s/2.7/config/recipes/beats/stack\_monitoring.yaml

I just want metrics and logs collected for elastic in one namespace, and the filebeat example provided sends all logs to my monitoring cluster, into a .ds-filebeat-8.7.0 index. This grows rapidly with other installations that are running in this cluster.

So, here's the working example to start with:

```auto
  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          scope: node
          hints:
            enabled: true
            default_config:
              type: container
              paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

```

I tried filtering on the namespace with a condition in the template, like so:

```auto
  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.default_config.enabled: "false"
          hints.enabled: "true"
          templates:
          - condition.equals.kubernetes.namespace: elastic
            config:
            - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
              type: container

```

I see in the filebeat logs that it is detecting the logs for the correct containers with the elastic namespace. I see that the .ds-filebeat index grows, but Kibana does not show these logs and it tells me that Beats were not detected.

Next, I tried adding the `co.elastic.logs/enabled: "true"` annotation to all my elastic and kibana pods. ECK appears to add another `co.elastic.logs/module: elasticsearch` annotation to the running pods and then I use this for the autodiscovery configuration:

```auto
  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          hints.default_config.enabled: "false"
          hints.enabled: "true"

```

I obtained this from this documentation section: [https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#\_kubernetes\_2](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#_kubernetes_2)  
Again, I see that in filebeat's logs, that it is detecting the pods with this annotation, but Kibana again does not pick up that these are elastic logs.

How do I configure filebeat autosdiscover to collect from one namespace and have Kibana pick this up? Is there a way to debug why Kibana is not picking up the logs in the "Stack Monitoring" view?

---

<div class="post-metadata">

**Author:** ![bdols](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bdols/32/121151_2.png) [@bdols](https://discuss.elastic.co/u/bdols)\
**Post date:** [May 18, 2023, 6:26am UTC](https://discuss.elastic.co/t/filebeat-kubernetes-autodiscovery-per-namespace-kibana-missing-beats/333722/2 "2023-05-18T06:26:44Z")

</div>

Since I can't add github links(?), here's a link to the link:

> **[Configuration Examples | Elastic Cloud on Kubernetes \[2.7\] | Elastic](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-beat-configuration-examples.html#k8s_elasticsearch_and_kibana_stack_monitoring)**

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 15, 2023, 8:27am UTC](https://discuss.elastic.co/t/filebeat-kubernetes-autodiscovery-per-namespace-kibana-missing-beats/333722/3 "2023-06-15T08:27:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.