Prometheus Federation Metricbeat - No index and document in ES

Hello,

We currently have a Kubernetes cluster that has a Prometheus Installation.

I'm looking to export the Prometheus gathered metrics to Elastic so we can create Kibana dashboard; using the Federation approach (so we do not have to change all annotations).

Here's the Kubernetes YAML.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-config
  namespace: monitoring
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    logging.level: info
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false
    processors:
      - add_cloud_metadata:
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: monitoring
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: prometheus
      enabled: true
      period: 180s
      metricsets: ["collector"]
      hosts: ["prometheus-kube-prometheus-prometheus:9090"]
      metrics_path: '/federate'
      headers: 
        accept: "text/plain"
      query:
        'match[]': '{__name__!=""}'
      timeout: 20s
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metricbeat
  namespace: monitoring
  labels:
    k8s-app: metricbeat
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: metricbeat
  template:
    metadata:
      labels:
        k8s-app: metricbeat
    spec:
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: metricbeat
        imagePullPolicy: Always
        image: docker.elastic.co/beats/metricbeat:7.12.0
        args: [
          "-c", "/etc/metricbeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: [redacted]
        - name: ELASTICSEARCH_PORT
          value: "9243"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: [redacted]
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-config
      - name: modules
        configMap:
          defaultMode: 0600
          name: metricbeat-deployment-modules
---

My problem is that metricbeat does not "seems" to send data to Elastic.

On Elastic, there's no metricbeat-* indices.

Here's the log of the metricbeat pod on Kubernetes

2021-04-20T14:01:27.220Z	INFO	instance/beat.go:660	Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2021-04-20T14:01:27.229Z	INFO	instance/beat.go:668	Beat ID: 975e3069-3277-4025-af26-1bf0231accf1
2021-04-20T14:01:27.236Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2021-04-20T14:01:27.236Z	INFO	[beat]	instance/beat.go:996	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "975e3069-3277-4025-af26-1bf0231accf1"}}}
2021-04-20T14:01:27.237Z	INFO	[beat]	instance/beat.go:1005	Build info	{"system_info": {"build": {"commit": "08e20483a651ea5ad60115f68ff0e53e6360573a", "libbeat": "7.12.0", "time": "2021-03-18T06:28:33.000Z", "version": "7.12.0"}}}
2021-04-20T14:01:27.237Z	INFO	[beat]	instance/beat.go:1008	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.15.8"}}}
2021-04-20T14:01:27.239Z	INFO	[beat]	instance/beat.go:1012	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-04-10T22:30:54Z","containerized":true,"name":"metricbeat-69f9758c46-dfmhj","ip":["127.0.0.1/8","10.4.1.115/24"],"kernel_version":"5.4.89+","mac":["d2:4a:61:eb:7f:f8"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"44ccc339d95bd51386fcfc5d8f041927"}}}
2021-04-20T14:01:27.239Z	INFO	[beat]	instance/beat.go:1041	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 6, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-04-20T14:01:26.820Z"}}}
2021-04-20T14:01:27.240Z	INFO	instance/beat.go:304	Setup Beat: metricbeat; Version: 7.12.0
2021-04-20T14:01:27.240Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'metricbeat-7.12.0' as ILM is enabled.
2021-04-20T14:01:27.241Z	INFO	eslegclient/connection.go:99	elasticsearch url: https://[redacted].gcp.elastic-cloud.com:9243
2021-04-20T14:01:27.243Z	INFO	[publisher]	pipeline/module.go:113	Beat name: metricbeat-69f9758c46-dfmhj
2021-04-20T14:01:27.247Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:105	add_cloud_metadata: hosting provider type detected as gcp, [redacted]
2021-04-20T14:01:27.269Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2021-04-20T14:01:27.269Z	INFO	instance/beat.go:468	metricbeat start running.
2021-04-20T14:01:27.271Z	INFO	cfgfile/reload.go:164	Config reloader started
2021-04-20T14:01:27.272Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
2021-04-20T14:01:37.353Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(elasticsearch(https://[redacted].gcp.elastic-cloud.com:9243))
2021-04-20T14:01:37.354Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-04-20T14:01:37.354Z	INFO	[publisher]	pipeline/retry.go:223	  done
2021-04-20T14:01:37.492Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.0
2021-04-20T14:01:37.496Z	INFO	[license]	licenser/es_callback.go:51	Elasticsearch license: Platinum
2021-04-20T14:01:37.498Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.0
2021-04-20T14:01:37.501Z	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
2021-04-20T14:01:37.504Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2021-04-20T14:01:37.504Z	INFO	[index-management]	idxmgmt/std.go:274	ILM policy successfully loaded.
2021-04-20T14:01:37.504Z	INFO	[index-management]	idxmgmt/std.go:407	Set setup.template.name to '{metricbeat-7.12.0 {now/d}-000001}' as ILM is enabled.
2021-04-20T14:01:37.504Z	INFO	[index-management]	idxmgmt/std.go:412	Set setup.template.pattern to 'metricbeat-7.12.0-*' as ILM is enabled.
2021-04-20T14:01:37.504Z	INFO	[index-management]	idxmgmt/std.go:446	Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.12.0 {now/d}-000001} as ILM is enabled.
2021-04-20T14:01:37.504Z	INFO	[index-management]	idxmgmt/std.go:450	Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-04-20T14:01:37.536Z	INFO	template/load.go:97	Template metricbeat-7.12.0 already exists and will not be overwritten.
2021-04-20T14:01:37.537Z	INFO	[index-management]	idxmgmt/std.go:298	Loaded index template.
2021-04-20T14:01:37.542Z	INFO	[index-management]	idxmgmt/std.go:309	Write alias successfully generated.
2021-04-20T14:01:37.544Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(elasticsearch(https://[redacted].gcp.elastic-cloud.com:9243)) established

I tried letting the pod runs for 1 hour, and nothing never gets to the Elastic Cluster.
I tried to curl the Prometheus from the metricbeat pod, and it works as intended.
I also tried to connect to the cluster using curl, and it works.