Metricbeat prometheus debug logging

I've gotten an autodiscover metricbeat for prometheus working (scraping a pod in k8s), but I keep getting the error:

unable to decode response from prometheus endpoint: decoding of metric family failed: text format parsing error in line 1: expected float as value, got "'t"

And I can't understand where it's getting the value 't from as it's nowhere in the /metrics response.

I can run Prometheus promtool against the /metrics path to verify that the prometheus format is valid.

I would like a way to gain insight into what metricbeat is trying to parse, but even setting maximum debug logging I'm not getting any info about which metric it's trying to parse...making it really hard to understand the problem. Is there a way to get verbose logging as to what values it's trying to parse? (or actually see the entire response it's trying to parse?). Doing a curl towards the /metrics endpoint from within the metricbeat pod I'm getting a valid I need to understand why metricbeats can't parse the response (or which response it's actually trying to parse).


I'm afraid this has to do with the text parsing and many things can go wrong (see similar

In order to investigate the problem through Metricbeat you should manually debug it from sourcecode. Here is where it fails, so you can most probably log the families that are successfully parsed so far and make out what is the one that cause the failure.


Thank you for the information! will probably help me out as well. Will try to see if I can build from source, or wait for 7.6.1 to come out.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.