Filebeat not collecting logs for hints based autodiscover in kubernetes

[sayantanvlabs (Sayantan Das)]

I am trying to get logs from pods annotated with co.elastic.logs/enabled: "true" into elasticsearch using filebeat. But it is not working. I am attaching the configurations I am using, please let me know if I am missing something.

# Setup Logstash.
cat <<EOF | kubectl apply -f -
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: jaegerpoc-elastic
  namespace: elastic-system
spec:
  count: 1
  elasticsearchRefs:
    - name: jaegerpoc-elastic
      clusterName: jes
  version: 8.9.1
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        output {
          elasticsearch {
            hosts => "http://jaegerpoc-elastic-es-http.elastic-system.svc:9200"
            user => "${JES_ES_USER}"
            password => "${JES_ES_PASSWORD}"
            index => "logs"
          }
        }
  services:
    - name: beats
      service:
        spec:
          type: ClusterIP
          ports:
            - port: 5044
              name: "filebeat"
              protocol: TCP
              targetPort: 5044
EOF

# Setup FileBeat
cat <<EOF | kubectl apply -f -
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: jaegerpoc-elastic
  namespace: elastic-system
spec:
  type: filebeat
  version: 8.9.1
  elasticsearchRef:
    name: jaegerpoc-elastic
  config:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: "minikube"
          hints:
            enabled: true
            default_config:
              enabled: false
              type: container
              paths:
                - "/var/log/containers/*\${data.kubernetes.container.id}.log"
    output.elasticsearch:
      enabled: false
    output.logstash:
      hosts: ["jaegerpoc-elastic-ls-beats.elastic-system.svc.cluster.local:5044"]
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: jaegerpoc-elastic-beat-sa
        automountServiceAccountToken : true
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
EOF

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jaegerpoc-elastic-beat-sa
  namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-beat-autodiscover
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  - apps
  resources:
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-beat-autodiscover-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elastic-beat-autodiscover
subjects:
- kind: ServiceAccount
  name: jaegerpoc-elastic-beat-sa
  namespace: elastic-system
EOF

Now my pods are being deployed with the logs enabled annotation. And I can confirm it is producing the logs.But it's not coming to elasticsearch.
Although if I use filebeat.inputs instead of autodiscover, I am getting logs for all the pods

filebeat.inputs:
    - type: container
      paths:
      - /var/log/containers/*.log