Metricbeat Promethes merging queries

[evileric77]

This has been mentioned before here:

Prometheus collector query defined once, applied everywhere

Hi, I'm using Metricbeat 8.6.2 and I'm trying to collect Prometheus metrics using the official Prom snmp exporter. This particular exporter requires passing a URL query string to configure the snmp endpoint to collect the metrics from. As an example, this exporter URL: http//snmpexporter:9116/snmp/?target=192.168.10.14&module=synology makes the snmp exporter return snmp metrics from IP 192.168.10.14 and the synology MIB module. Here is a sample configuration which collects Prometheus metrics …

But as there is no resolution there I'm posting here and will open an issue on github shortly. With the Prometheus module if you define 2 items that leverage the module, metricbeat (as a result of composing it seems) merges it and causes invalid queries to result. In my case this is scraping a rook-ceph instance and a custom vsphere exporter:

prometheus-rook-ceph.yml

    # Module: prometheus
    # Docs: https://www.elastic.co/guide/en/beats/metricbeat/8.4/metricbeat-metricset-prometheus-remote_write.html
    - module: prometheus
      metrics_path: /metrics
      period: 30s
      hosts: ["rook-ceph-mgr.rook-ceph:9283"]
      tags: ["prometheus"]
      processors:
      - add_fields:
          target: ''
          fields:
            type: 'ceph-prometheus'
            site: "${SITE}"

prometheus-vmware-exporter.yml

    # /usr/share/metricbeat/modules.d/
    # Module: prometheus
    # Docs: https://www.elastic.co/guide/en/beats/metricbeat/8.4/metricbeat-metricset-prometheus-remote_write.html
    - module: prometheus
      metrics_path: /metrics
      query: 
        vsphere_host: "dev-esxi01.mydomain.net"
      period: 30s
      hosts: ["vmware-exporter:9272"]
      tags: ["prometheus"]
      processors:
      - add_fields:
          target: ''
          fields:
            type: 'vmware-exporter'
            site: "${SITE}"

The expected behavior is that metricbeat scrapes each endpoint individually. What actually happens is the configs are errantly merged in some way resulting in errantly querying the bare /metrics endpoint for ceph mgr:
http://rook-ceph-mgr.rook-ceph:9283/metrics?vsphere_host=dev-esxi01.mydomain.net

[evileric77]

After looking at the code this seems more related to the use of global vars and pointers in the metricbeat prometheus module itself. I'm reading through the code more but I do believe this is an issue that stems from the way the code in Go is written. beats/collector.go at main · elastic/beats · GitHub

[evileric77]

I guess this module doesn't get much use since there is 0 traction in here regarding this issue from myself and one other person. Probably solved many times via workaround with multiple metricbeat instances running.