I've gotten an autodiscover metricbeat for prometheus working (scraping a pod in k8s), but I keep getting the error:
unable to decode response from prometheus endpoint: decoding of metric family failed: text format parsing error in line 1: expected float as value, got "'t"
And I can't understand where it's getting the value 't from as it's nowhere in the /metrics response.
I can run Prometheus promtool against the /metrics path to verify that the prometheus format is valid.
I would like a way to gain insight into what metricbeat is trying to parse, but even setting maximum debug logging I'm not getting any info about which metric it's trying to parse...making it really hard to understand the problem. Is there a way to get verbose logging as to what values it's trying to parse? (or actually see the entire response it's trying to parse?). Doing a curl towards the /metrics endpoint from within the metricbeat pod I'm getting a valid response...so I need to understand why metricbeats can't parse the response (or which response it's actually trying to parse).