Not able to parse prometheus metrics to json format

Elastic Stack Logstash
21

Same 401 unauthorised issue after below config:

metricbeat.modules:
- module: prometheus
  period: 10s
  hosts: ["http://testserver.dev.com:42115"]
  metrics_path: "/rpsservice/actuator/prometheus?tId=e5bf6fc7-457a-406f-b2df-1a38be998f51"
  headers:
    Authorization: "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjViOWM3N2FiLTY4OTktNDlhNC1hYzA5LWMxYzE2OWNjMGQ0NCJ9.eyJzdWIiOiJiZTUwYmZkMS1hOGQ4LTRmZjItOGQ0YS0wNTQzZDY0OWJmMjUiLCJncmFudF90eXBlIjoiY2xpZW50X2NyZWRlbnRpYWxzIiwidG9rZW5fdXNlIjpudWxsLCJzY29wZSI6IkNVU1RPTS1TQ09QRVMiLCJpc3MiOiJFVEYiLCJleHAiOjE3Mzk0MjYwMTEsImlhdCI6MTczOTQyMjQxMSwiY2xpZW50X2lkIjoiYmU1MGJmZDEtYThkOC00ZmYyLThkNGEtMDU0M2Q2NDliZjI1IiwidElkIjoiZTViZjZmYzctNDU3YS00MDZmLWIyZGYtMWEzOGJlOTk4ZjUxIiwianRpIjoiYWE0MGY5ZjYtYWMxNy00ZjMyLWFlZTAtOTk5ZjA1ZDZlN2NhIn0.NPsNP31m4x6E4FQxaPHVgvGvKlWvE671WBP2txTlUVH4X1NGBAUn7y1Po4YNa63751--"
22

Not sure what to tell you....

Did you run metricbeat with the -e -d "*" and carefully inspect the output...

Is there any other meaningful output . You are not actually sharing error logs...

What version of metricbeat?

I have not encountered same issue I just set up and exporter and connected first time.. I don't have it secured.. but these are common connections options ..

Ohh try these

metricbeat test config
metricbeat test prometheus 
metricbeat test output
23

Error Logs:

{"log.level":"debug","@timestamp":"2025-02-13T06:06:06.081+0530","log.logger":"logstash","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/logstash.(*asyncClient).Publish","file.name":"logstash/async.go","file.line":172},"message":"33 events out of 33 events sent to logstash host localhost:5044. Continue sending","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-02-13T06:06:06.088+0530","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).handleBatchSig","file.name":"memqueue/ackloop.go","file.line":80},"message":"ackloop: return ack to broker loop:33","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-02-13T06:06:06.089+0530","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).handleBatchSig","file.name":"memqueue/ackloop.go","file.line":82},"message":"ackloop:  done send ack","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-02-13T06:06:12.022+0530","log.logger":"prometheus.collector","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/helper/prometheus.(*prometheus).GetFamilies","file.name":"prometheus/prometheus.go","file.line":105},"message":"error received from prometheus endpoint: ","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-02-13T06:06:12.023+0530","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError","file.name":"module/wrapper.go","file.line":333},"message":"Error fetching data for metricset prometheus.collector: unable to decode response from prometheus endpoint: unexpected status code 401 from server","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-02-13T06:06:16.082+0530","log.logger":"logstash","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/logstash.(*asyncClient).Publish","file.name":"logstash/async.go","file.line":172},"message":"32 events out of 32 events sent to logstash host localhost:5044. Continue sending","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-02-13T06:06:16.091+0530","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).handleBatchSig","file.name":"memqueue/ackloop.go","file.line":80},"message":"ackloop: return ack to broker loop:32","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-02-13T06:06:16.092+0530","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).handleBatchSig","file.name":"memqueue/ackloop.go","file.line":82},"message":"ackloop:  done send ack","service.name":"metricbeat","ecs.version":"1.6.0"}

Version:

metricbeat-8.17.2-linux-x86_64

24 
[root@Testserver metricbeat-8.17.2-linux-x86_64]# ./metricbeat test config
Config OK


[root@Testserver metricbeat-8.17.2-linux-x86_64]# ./metricbeat test output
logstash: localhost:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: ::1, 127.0.0.1
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
25

And this one

26 
[root@Testserver metricbeat-8.17.2-linux-x86_64]# ./metricbeat test prometheus
Test config

Usage:
  metricbeat test [command]

Available Commands:
  config      Test configuration settings
  modules     Test modules settings
  output      Test metricbeat can connect to the output by using the current settings

Flags:
  -h, --help   help for test

Global Flags:
  -E, --E setting=value              Configuration overwrite
  -c, --c string                     Configuration file, relative to path.config (default "metricbeat.yml")
  -d, --d string                     Enable certain debug selectors
  -e, --e                            Log to stderr and disable syslog/file output
      --environment environmentVar   set environment being ran in (default default)
      --path.config string           Configuration path
      --path.data string             Data path
      --path.home string             Home path
      --path.logs string             Logs path
      --plugin pluginList            Load additional plugins
      --strict.perms                 Strict permission checking on config files (default true)
  -v, --v                            Log at INFO level

Use "metricbeat test [command] --help" for more information about a command.
27

Ignore

28 
prometheus...
  collector...OK
    result:
    {
     "@timestamp": "2025-02-13T00:55:41.402Z",
     "event": {
      "dataset": "prometheus.collector",
      "duration": 10621946,
      "module": "prometheus"
     },
     "metricset": {
      "name": "collector",
      "period": 10000
     },
     "prometheus": {
      "labels": {
       "instance": "Testserver.dev.com:42115",
       "job": "prometheus"
      },
      "metrics": {
       "up": 0
      }
     },
     "service": {
      "address": "http://Testserver.dev.com:42115/rpsservice/actuator/prometheus%3FtId=e5bf6fc7-457a-406f-b2df-1a38be998f51",
      "type": "prometheus"
     }
    }

If we observe service address, http url is different than what i had put in there. "?" in path is converting to %3F. Might be causing issue?

29

./metricbeat test modules prometheus

I think that is the same.

%3F is URL encode of ? so in don't think that is the issue

Looks like is connected to the base path fine / authenticated so I don't think it is having problem with authentication

So I think your endpoint is not returning prometheus metrics as expected format.. Because that looks like it connected to the base path fine..

30

If you see the error log, it says 401 status returned which is unauthorized, right

31

Yup ..

Also says

unable to decode response so perhaps 401 is misleading

Not sure what to tell you never ran into this.. I don't think what is being returned is metrics format ... We have literally 1000s of users that use prometheus module

I think your actuator endpoint does not provide in /metrics format

32

EDIT : Took out wont work...

Question are these micrometer metrics?

Apologies I couldn't solve this

33

Yeah those looks like prometheus

Mine

node_cpu_seconds_total{cpu="13",mode="idle"} 261376.83
node_cpu_seconds_total{cpu="13",mode="nice"} 0
node_cpu_seconds_total{cpu="13",mode="system"} 656.75
node_cpu_seconds_total{cpu="13",mode="user"} 921.48
node_cpu_seconds_total{cpu="14",mode="idle"} 245167.35
....
node_filesystem_avail_bytes{device="/dev/disk1s1s1",device_error="",fstype="apfs",mountpoint="/"} 6.5457954816e+11
node_filesystem_avail_bytes{device="/dev/disk1s2",device_error="",fstype="apfs",mountpoint="/System/Volumes/Data"} 6.5457954816e+11
node_filesystem_avail_bytes{device="/dev/disk1s3",device_error="",fstype="apfs",mountpoint="/System/Volumes/Preboot"} 6.54579499008e+11
node_filesystem_avail_bytes{device="/dev/disk1s5",device_error="",fstype="apfs",mountpoint="/System/Volumes/VM"} 6.54585479168e+11
node_filesystem_avail_bytes{device="/dev/disk1s6",device_error="",fstype="apfs",mountpoint="/System/Volumes/Update"} 6.54585479168e+11
node_filesystem_avail_bytes{device="/dev/disk3s1",device_error="",fstype="apfs",mountpoint="/private/tmp/dmg.opkzUX"} 8.4058112e+07
node_filesystem_avail_bytes{device="map auto_home",device_error="",fstype="autofs",mountpoint="/System/Volumes/Data/home"} 0

not sure what to tell ya ... I am baffled...

34

Below are mine, @stephenb No worries, thanks for your time and patience. Will keep trying and let you know if solved

 HELP cache_puts_total The number of entries added to the cache
# TYPE cache_puts_total counter
cache_puts_total{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="APPCONFIG_CACHE",cache_manager="cacheManager",host="in7lin059",name="APPCONFIG_CACHE"} 0.0
cache_puts_total{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="RPSConfiguration",cache_manager="cacheManager",host="in7lin059",name="RPSConfiguration"} 37.0
cache_puts_total{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="RPSConfigurationV2",cache_manager="cacheManager",host="in7lin059",name="RPSConfigurationV2"} 739.0
cache_puts_total{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="ftaCodeCache",cache_manager="cacheManager",host="in7lin059",name="ftaCodeCache"} 0.0
cache_puts_total{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="ivaOperatorCache",cache_manager="cacheManager",host="in7lin059",name="ivaOperatorCache"} 0.0
cache_puts_total{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="tenantInfoCache",cache_manager="cacheManager",host="in7lin059",name="tenantInfoCache"} 0.0
# HELP cache_removals Cache removals
# TYPE cache_removals gauge
cache_removals{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="APPCONFIG_CACHE",cache_manager="cacheManager",host="in7lin059",name="APPCONFIG_CACHE"} 0.0
cache_removals{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="RPSConfiguration",cache_manager="cacheManager",host="in7lin059",name="RPSConfiguration"} 24.0
cache_removals{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="RPSConfigurationV2",cache_manager="cacheManager",host="in7lin059",name="RPSConfigurationV2"} 336.0
cache_removals{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="ftaCodeCache",cache_manager="cacheManager",host="in7lin059",name="ftaCodeCache"} 0.0
cache_removals{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="ivaOperatorCache",cache_manager="cacheManager",host="in7lin059",name="ivaOperatorCache"} 0.0
cache_removals{CUST="RPS",ENV="DEV",PRODUCT="GTM",cache="tenantInfoCache",cache_manager="cacheManager",host="in7lin059",name="tenantInfoCache"} 0.0
35

Heya @Bhanu_Praveen

Not sure this will help much, but I remembered this thread...

It turns out I had another user who was doing Spring Boot, Micrometer + Actuator.

I got it to work.

I used this as a sample app.

Here is my metricbeat / agent config.

- module: prometheus
  period: 10s
  hosts: ["localhost:8080"]
  metrics_path: /actuator/prometheus
  use_types: true
  rate_counters: true

Screenshot 2025-02-23 at 11.15.39 AM
Screenshot 2025-02-23 at 11.15.39 AM2148×1484 222 KB

Percentiles... no authorization... but this definitely works... I think your issue is probably around that extra parameters but I don't know why

Screenshot 2025-02-23 at 11.17.15 AM
Screenshot 2025-02-23 at 11.17.15 AM2336×670 139 KB

I also got it to work with the Java APM Agent Directly