Metricbeat prometheus module collector metricset escaping

Hi, i'm using metricbeat 7.10.1 and i want to get data with prometheus module(golang prometheus client) and send data to kafka output.
Here's my yaml file.

  metricbeat.yml: |-
    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          include_annotations: ["prometheus.io.scrape"]
          node: ${NODE_NAME}
          templates:
            - condition:
                contains:
                  kubernetes.annotations.prometheus.io/scrape: "true"
              config:
                - module: prometheus
                  metricsets: ["collector"]
                  hosts: "${data.host}:${data.port}"
                  period: 30s
                  query:
                    format: prometheus
                  metrics_path: /api/v1/test/metrics
                  fields:
                    topic: test_apm-prom
                    host_type: kubernetes
                  service.name: test
    output.kafka:
      hosts: ["x.x.x.x.:9092"]
      topic: 'test_topic'
      partition.round_robin:
        reachable_only: false
      required_acks: 1
      compression: gzip
      max_message_bytes: 1000000

My question is if i using metricsets: ["collector"], my mericbeat request /api/v1/test/metrics, and it return this result:

promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
go_gc_duration_seconds_sum 0.063136861
go_gc_duration_seconds_count 1487
go_goroutines 12
go_memstats_alloc_bytes 4.780744e+06

and here is problem.
My return data contains { and }, so when i use output.kafka, data seperated and metricbeat send data to kafka several times.
Is there any way to escape these { and } ?

Hi @4orty,

Do you mean that the same data is duplicated? or that Metricbeat sends multiple events for each request to prometheus?

It is expected that Metricbeat sends multiple events for each request to Prometheus, one for every set of labels (the values between { and }).

So for example, for the case you mention I would expect at least three events:

  • One for metrics with code="500" (they will contain a field prometheus.labels.code: 500).
  • Another one for metrics with code="503" (they will contain a field prometheus.labels.code: 503).
  • And a last one for metrics without labels.