Filebeats autodiscover on kubernetes doesn't collect logs from annotated pods

[Ivan_Kovacic (Ivan Kovačić)]

Hi,
I have an issue with filebeat when trying to collect logs from annotated pods, i have been following the documentation but without success. I am using filebeats 7.17 on kubernetes v1.21.5.
This is my filebeat configuration

      filebeat.autodiscover:
        providers:
          - type: kubernetes
            hints.enabled: true
            hints.default_config.enabled: false

      # filebeat.inputs:
      #   - type: container
      #     multiline:
      #       type: pattern
      #       pattern: '^\x1b\[[0-9;]+m'
      #       negate: true
      #       match: after
      #     paths:
      #       - /var/log/container/*-689-*.log
      #     processors:
      #       - add_kubernetes_metadata:
      #           host: ${NODE_NAME}
      #           matchers:
      #             - logs_path:
      #                 logs_path: "/var/log/containers/"
      output.logstash:
        hosts: ["logstash-logstash-headless.logging:5044"]

and i have annotated the pods i want to harvest logs from with

co.elastic.logs/enabled: "true"

When i tried with autodiscover with default_config.enabled as false, i can see that filebeats registers node, but all i get is "Non-zero metrics in the last 30s" and nothing can be seen from kibana.

If I try with autodiscover, but with using templates i can get the logs in kibana, same as if i go with filebeat.inputs for container (the code comented out). Only that way is tedious to maintain, since every newly created pod (whether a new project or w/e, the filebeat config needs to be updated).

Am i doing something obviously wrong and can't pinpoint it, or is the functionality broken?