Metricbeat Prometheus module not support match query

Elastic Stack Beats
1

Metricbeat: 7.0.0

config:
metricbeat.modules:

  • module: prometheus
    period: 10s
    hosts: ["<prometheus_url>"]
    metrics_path: '/federate'
    query:
    'match[]': '{name!=""}'
    namespace: example

This will result in:
"error": {
"message": "Unable to decode response from prometheus endpoint"
}

is this a bug?

config quotes from:

Dec 23rd, 2018: [EN][Observability] Querying metrics from Prometheus with Elastic Stack

2

The query works in a curl though.

3

Looks like this issue is due to the federate API returned too many events. is there a limit？

I further verified that when a query in a Prometheus directive returns over roughly 200k events, the error comes out. Had tuned the queue.mem.events and output.elasticsearch bulk_max_size, both have no help on this case.

4

Thank you for your feedback @Jehutywong!

This sounds like a limit set somewhere in the client we use to scrape Prometheus metrics. Could you please open a new issue for it? https://github.com/elastic/beats/issues/new/choose

As a workaround, you could limit the number of metrics you get by updating the query to match only a subset.

Best regards

5

Thanks @exekias

I have a case opened in github #11912

Cut query into pieces works but hard to make sure all subsets not over limit.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.