Metricbeat Prometheus module not support match query

[Jehutywong]

Metricbeat: 7.0.0

config:
metricbeat.modules:

• module: prometheus
  period: 10s
  hosts: ["<prometheus_url>"]
  metrics_path: '/federate'
  query:
  'match[]': '{name!=""}'
  namespace: example

This will result in:
"error": {
"message": "Unable to decode response from prometheus endpoint"
}

is this a bug?

config quotes from:

Dec 23rd, 2018: [EN][Observability] Querying metrics from Prometheus with Elastic Stack

[Jehutywong]

The query works in a curl though.

[Jehutywong]

Looks like this issue is due to the federate API returned too many events. is there a limit？

I further verified that when a query in a Prometheus directive returns over roughly 200k events, the error comes out. Had tuned the queue.mem.events and output.elasticsearch bulk_max_size, both have no help on this case.

[exekias (Carlos Pérez Aradros)]

Thank you for your feedback @Jehutywong!

This sounds like a limit set somewhere in the client we use to scrape Prometheus metrics. Could you please open a new issue for it? https://github.com/elastic/beats/issues/new/choose

As a workaround, you could limit the number of metrics you get by updating the query to match only a subset.

Best regards

[Jehutywong]

Thanks @exekias

I have a case opened in github #11912

Cut query into pieces works but hard to make sure all subsets not over limit.