Kubernetes autodiscover template not working

[rodriguez-facundo (Facundo Luis Rodriguez)]

I'm trying to use kuberntes autodiscover templating to collect metrics from a subset of pods, but I am not able to get output unless I remove the template and set hint.enabled=true

// metricbeat.yaml
      providers:
        - type: kubernetes
          host: ${HOSTNAME}   // also tried  ${NODE_NAME}
          include_annotations: ["metribeat"]
          templates:
            - condition.contains:
                kubernetes.annotations.metribeat: "true"
              config:
                - module: kubernetes
                  metricsets: ["node", "system","pod","container","volume"]
                  period: 10s
                  hosts: ["localhost:10255"]

The pod:

annotations:
        metricbeat: "true"

system.yaml

  system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5
        by_memory: 5

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

I'm commenting this bit (which works just fine removing the templating and using hint.enabled=true but it collects from everything in the cluster and it became super expensive):

# kubernetes.yml: |-
  #   - module: kubernetes
  #     metricsets:
  #       - node
  #       - system
  #       - pod
  #       - container
  #       - volume
  #     period: 10s
  #     host: ${NODE_NAME}
  #     hosts: ["localhost:10255"]
    
  #   - module: kubernetes
  #     enabled: true
  #     metricsets:
  #       - state_node
  #       - state_deployment
  #       - state_replicaset
  #       - state_statefulset
  #       - state_pod
  #       - state_container
  #     period: 10s
  #     hosts: ["kube-state-metrics.kube-system.svc.cluster.local:8080"]
  #     add_metadata: true
  #     in_cluster: true