Not able to parse prometheus metrics to json format

Elastic Stack Logstash
1

Hello,

Able to get Prometheus metrics as a message using http_poller. But not able to convert to json.

Below is the same output of Prometheus format

# TYPE http_requests_total counter 
http_requests_total{code="200",method="GET"} 28 
http_requests_total{code="200",method="POST"} 3```

Looking to get into below format if possible:

{
    "Metrics": [
        {
            "http_requests_total": {
                "code": "200",
                "method": "GET",
                "value": 28
            }
        },
        {
            "http_requests_total": {
                "code": "200",
                "method": "POST",
                "value": 3
            }
        }
    ]
}

is there any other way i can get the expected output in json pls let me know?

2

What exactly are you using to do the polling?
metricbeat / elastic agent?
What version?
Example are you using metricbeat prometheus module it should do this for you... .

3

As i do not want to install Metricbeat on all prod systems, I am using logstash http_poller input as below and able to get the data:

input {
   http_poller {
      urls => 
		{
         rps => 
		 {
			 url => "http://fsdfsdf.324.com"
			headers => 
			{
				"Authorization" => "Bearer adad.dad.da-0c1FqzERCL-ad-adsa-uVdghmYZA0A"
			}
		}
      }
	  keepalive => true
      automatic_retries => 1
      schedule => { cron => "* * * * * UTC"}
      #codec => "json"
	  metadata_target => "http_poller_metadata"
   }
     
}
4

You don't need to install metricbeat on every host...

Just install metricbeat somewhere and use the endpoint ... it can be on a central server... basically the same as you are doing with logstash...

So just install metricbeat on your logstash host and give it a try.

depending on scale you might need more than one..

- module: prometheus
  period: 10s
  hosts: ["http://fsdfsdf.324.com:9100"]
  metrics_path: /metrics
  #metrics_filters:
  #  include: []
  #  exclude: []
  #username: "user"
  #password: "secret"

If you just do what you are doing above you will need to do all the prometheus parsing and transformation in logstash which will be a lot... a lot ... did I mention ... a LOT of work that metricbeat already does....

Give it a try...

5

Thanks, i will try metricbeat. Pls let me know how do i put a bearer token authorization which is required for metric endpoint access?

6

I suggest you look at the docs...

  # This can be used for service account based authorization:
  #bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  #ssl.certificate_authorities:
  #  - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
7

Tried with below config:

metricbeat.modules:
- module: prometheus
  period: 10s
  hosts: ["http://dasda.xde.com:42115"]
  metrics_path: /adaw/actuator/prometheus?tId=df-1a38be998f51
  bearer_token_file: /opt/metricbeat-8.17.2-linux-x86_64/token.txt

Getting unauthorised response as below:

{"log.level":"error","@timestamp":"2025-02-13T03:38:11.183+0530","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError","file.name":"module/wrapper.go","file.line":333},"message":"Error fetching data for metricset prometheus.collector: unable to decode response from prometheus endpoint: unexpected status code 401 from server","service.name":"metricbeat","ecs.version":"1.6.0"}

Any help around here pls? Thinking token is not taking in properly.

8

401 is authorization...

Unfortunately, bearer token for this is not my area expertise... but common mistake I see.

Is it properly encoded?
Did you add an extra new line (pretty common mistake)?
Did you try a curl with the same token?

As the docs say you can also use any other standard HTTP setting

So you could try headers as well

headers:
  Authorization: "Bearer mytoken123"
9

ya tried in post man with same token and working fine. Raised a Q in beats. Thanks for your support.

10

Try the headers option that should be equivalent of POST man -H

11

Tried and same response..

12

Did you base64 encode it?

13

postman generate in base64 only. In logstash below config working for me:

			headers => 
			{
				"Authorization" => "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ijk0MGRjNzBkLTUyM2MtNDNlOS05YzZkLWI2ZWU3MDMxOWYzMyJ9.eyJzdWIiOiJiZTUwYmZkMS1hOGQ4LTRmZjItOGQ0YS0wNTQzZDY0OWJmMjUiLCJncmFudF90eXBlIjoiY2xpZW50X2NyZWRlbnRpYWxzIiwidG9rZW5fdXNlIjpudWxsLCJzY29wZSI6IkNVU1RPTS1TQ09QRVMiLCJpc3MiOiJFVEYiLCJleHAiOjE3Mzk0MTc5NjIsImlhdCI6MTczOTQxNDM2MiwiY2xpZW50X2lkIjoiYmU1MGJmZDEtYThkOC00ZmYyLThkNGEtMDU0M2Q2NDliZjI1IiwidElkIjoiZTViZjZmYzctNDU3YS00MDZmLWIyZGYtMWEzOGJlOTk4ZjUxIiwianRpIjoiMGQ4NjM4ZTktMzE1OS00YmYzLTg1MGEtZjVhZWM4MTU3NDYwIn0.MzNr7h5JJnNeMJYAKNUIaJn_Vvv2x3qkFxGMTsYh-ZA9d3FXOVN0cHFhKeRu5pxIRPutabDTOz_fFh9HXEbXhqp7ZM8Nmh56VJ8N7qe8lg5eBSsNF5pt3-nxLNS81Qw5Wc50vRVre74KGaOtFpMQjU7BKBnHZtJgii5---paCjw0Bc7uFiPoEoAase6D5nY2bz8Jq3y-7PaBBmoYUAmgraTd8IlVfb07ULSA"
			}

Same token not working in beats. I am saving token file as txt, hope that is OK?

Below is the token.txt file content:

"Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ijk0MGRjNzBkLTUyM2MtNDNlOS05YzZkLWI2ZWU3MDMxOWYzMyJ9.eyJzdWIiOiJiZTUwYmZkMS1hOGQ4LTRmZjItOGQ0YS0wNTQzZDY0OWJmMjUiLCJncmFudF90eXBlIjoiY2xpZW50X2NyZWRlbnRpYWxzIiwidG9rZW5fdXNlIjpudWxsLCJzY29wZSI6IkNVU1RPTS1TQ09QRVMiLCJpc3MiOiJFVEYiLCJleHAiOjE3Mzk0MTc5NjIsImlhdCI6MTczOTQxNDM2MiwiY2xpZW50X2lkIjoiYmU1MGJmZDEtYThkOC00ZmYyLThkNGEtMDU0M2Q2NDliZjI1IiwidElkIjoiZTViZjZmYzctNDU3YS00MDZmLWIyZGYtMWEzOGJlOTk4ZjUxIiwianRpIjoiMGQ4NjM4ZTktMzE1OS00YmYzLTg1MGEtZjVhZWM4MTU3NDYwIn0.MzNr7h5JJnNeMJYAKNUIaJn_Vvv2x3qkFxGMTsYh-ZA9d3FXOVN0cHFhKeRu5pxIRPutabDTOz_fFh9HXEbXhqp7ZM8Nmh56VJ8N7qe8lg5eBSsNF5pt3-nxLNS81Qw5Wc50vRVre74KGaOtFpMQjU7BKBnHZtJgii5-LuwPO0NOOGaUnBnvZ8FCkCvG8M_dCl--paCjw0Bc7uFiPoEoAase6D5nY2bz8Jq3y-7PaBBmoYUAmgraTd8IlVfb07ULSA"
14

Don't put the word Bearer in the token file just the token... Not sure if quotes or not are needed

Hmmm should not be this hard....

15

Tried all 3 options, without Bearer, removing quotes, with quotes etc..No luck :thinking:

16

So you are hitting the actuator endpoint... Is that what you want ... Not the metrics endpoint?

Something Basic going on..

Go back to using the header option...

And try the /metrics endpoint

Are you using the correct port?

Start metricbeat like this

./metricbeat -e -d "*"

And check the logs output ... There will be a lot

See if you get more information

Can you curl without postman from this same host using curl -H Does it work?

Under the covers metricbeat does the same

If curl works from the same host metricbeat should / will

curl http://endpoint:port/metrics -H "Authorization: Bearer token"

BTW is it http or https?

Something basic going on ...

17

Below is cur config, so metrics path, i shd change to /metrics? it will not work

metricbeat.modules:
- module: prometheus
  period: 10s
  hosts: ["http://testserver.dev.com:42115"]
  metrics_path: /rpsservice/actuator/prometheus?tId=e5bf6fc7-457a-406f-b2df-1a38be998f51
 #bearer_token_file: /opt/hc/metricbeat-8.17.2-linux-x86_64/token.txt
  headers:
    Authorization: "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjViOWM3N2FiLTY4OTktNDlhNC1hYzA5LWMxYzE2OWNjMGQ0NCJ9.eyJzdWIiOiJiZTUwYmZkMS1hOGQ4LTRmZjItOGQ0YS0wNTQzZDY0OWJmMjUiLCJncmFudF90eXBlIjoiY2xpZW50X2NyZWRlbnRpYWxzIiwidG9rZW5fdXNlIjpudWxsLCJzY29wZSI6IkNVU1RPTS1TQ09QRVMiLCJpc3MiOiJFVEYiLCJleHAiOjE3Mzk0MjYwMTEsImlhdCI6MTczOTQyMjQxMSwiY2xpZW50X2lkIjoiYmU1MGJmZDEtYThkOC00ZmYyLThkNGEtMDU0M2Q2NDliZjI1IiwidElkIjoiZTViZjZmYzctNDU3YS00MDZmLWIyZGYtMWEzOGJlOTk4ZjUxIiwianRpIjoiYWE0MGY5ZjYtYWMxNy00ZjMyLWFlZTAtOTk5ZjA1ZDZlN2NhIn0.NPsNP31m4x6E4FQxaPHVgvGvKlWvE671WBP2txTlUVH4X1NGBAUn7y1Po4YNa63751-Y0MmiducdUak19twB41ltIo062BJVFghN9aRA5GQhwRvOxQlWKeNRXPwb_9exu3mR2KdjPh7uXSqXlC3Uh56fSuTtH_e_ib4ulyUsRMdpvzPHd36dujnEcp5fp4wvGUw7bzCxlNzK4B3OHpViSVUSuBwhdBKAmYHUJoLdZqvllHBi3PQFJzktE-"
18

My point is run curl from command line and get that to work ..

So run curl with your endpoint... Get curl to work

The we will configure metricbeat

I am not at your keyboard to debug... Try all options

You might need to put your metric_path value in quotes since it is not simple

curl "http://testserver.dev.com:42115/disservice/actuator/prometheus?tId=e5bf6fc7-457a-406f-b2df-1a38be998f51" -H "Authorization: Bearer token"

19

Curl is working just fine with below cmd, able to get all the data


curl http://testserver.dev.com:42115/rpsservice/actuator/prometheus?tId=e5bf6fc7-457a-406f-b2df-1a38be998f51 -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjViOWM3N2FiLTY4OTktNDlhNC1hYzA5LWMxYzE2OWNjMGQ0NCJ9..NPsNP31m4x6E4FQxaPHVgvGvKlWvE671WBP2txTlUVH4X1NGBAUn7y1Po4YNa63751-Y0MmiducdUak19twB41ltIo062BJVFghN9aRA5GQhwRvOxQlWKeNRXPwb_9exu3mR2KdjPh7uXSqXlC3Uh56fSuTtH_e_ib4ulyUsRMdpvzPHd36dujnEcp5fp4wvGUw7bzCxlNzK4B3OHpViSVUSuBwhdBKAmYHUJoLdZqvllHBi3PQFJzktE-wT7fjSztfYYP3XHioJFwKXQfEm8Q1r0yDGDTOLznxLiXw1_PDiBMqoovx3ige133mcQ4_okIyQLNbaFNuTSEMwPPPIHA"
20

Good

Put the metrics_path in "s