# Not able to parse prometheus metrics to json format **URL:** \ **Category:** Logstash\ **Created:** [February 12, 2025, 5:56pm UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447 "2025-02-12T17:56:57Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 12, 2025, 5:56pm UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/1 "2025-02-12T17:56:57Z")
Hello, Able to get Prometheus metrics as a message using http\_poller. But not able to convert to json. Below is the same output of Prometheus format ````auto # TYPE http_requests_total counter http_requests_total{code="200",method="GET"} 28 http_requests_total{code="200",method="POST"} 3``` ```` Looking to get into below format if possible: ```auto { "Metrics": [ { "http_requests_total": { "code": "200", "method": "GET", "value": 28 } }, { "http_requests_total": { "code": "200", "method": "POST", "value": 3 } } ] } ``` is there any other way i can get the expected output in json pls let me know? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 12, 2025, 6:30pm UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/2 "2025-02-12T18:30:34Z")
What exactly are you using to do the polling? metricbeat / elastic agent? What version? Example are you using metricbeat prometheus module it should do this for you... . > **[Prometheus module | Metricbeat Reference \[8.17\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-prometheus.html)** ---
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 13, 2025, 1:26am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/3 "2025-02-13T01:26:03Z")
As i do not want to install Metricbeat on all prod systems, I am using logstash http\_poller input as below and able to get the data: ```auto input { http_poller { urls => { rps => { url => "http://fsdfsdf.324.com" headers => { "Authorization" => "Bearer adad.dad.da-0c1FqzERCL-ad-adsa-uVdghmYZA0A" } } } keepalive => true automatic_retries => 1 schedule => { cron => "* * * * * UTC"} #codec => "json" metadata_target => "http_poller_metadata" } } ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 13, 2025, 1:54am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/4 "2025-02-13T01:54:23Z")
> [@Bhanu\_Praveen](#): > > As i do not want to install Metricbeat on all prod systems, You don't need to install metricbeat on every host... Just install metricbeat somewhere and use the endpoint ... it can be on a central server... basically the same as you are doing with logstash... So just install metricbeat on your logstash host and give it a try. depending on scale you might need more than one.. ```auto - module: prometheus period: 10s hosts: ["http://fsdfsdf.324.com:9100"] metrics_path: /metrics #metrics_filters: # include: [] # exclude: [] #username: "user" #password: "secret" ``` If you just do what you are doing above you will need to do all the prometheus parsing and transformation in logstash which will be a lot... a lot ... did I mention ... **a LOT** of work that metricbeat already does.... Give it a try... ---
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 13, 2025, 2:18am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/5 "2025-02-13T02:18:05Z")
Thanks, i will try metricbeat. Pls let me know how do i put a bearer token authorization which is required for metric endpoint access? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 13, 2025, 2:25am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/6 "2025-02-13T02:25:51Z")
I suggest you look at the docs... > **[Prometheus collector metricset | Metricbeat Reference \[8.17\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-prometheus-collector.html)** ```auto # This can be used for service account based authorization: #bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token #ssl.certificate_authorities: # - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ``` ---
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 13, 2025, 3:11am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/7 "2025-02-13T03:11:16Z")
Tried with below config: ```auto metricbeat.modules: - module: prometheus period: 10s hosts: ["http://dasda.xde.com:42115"] metrics_path: /adaw/actuator/prometheus?tId=df-1a38be998f51 bearer_token_file: /opt/metricbeat-8.17.2-linux-x86_64/token.txt ``` Getting unauthorised response as below: ```auto {"log.level":"error","@timestamp":"2025-02-13T03:38:11.183+0530","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError","file.name":"module/wrapper.go","file.line":333},"message":"Error fetching data for metricset prometheus.collector: unable to decode response from prometheus endpoint: unexpected status code 401 from server","service.name":"metricbeat","ecs.version":"1.6.0"} ``` Any help around here pls? Thinking token is not taking in properly. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 13, 2025, 3:22am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/8 "2025-02-13T03:22:40Z")
> [@Bhanu\_Praveen](#): > > `code 401 ` 401 is authorization... Unfortunately, bearer token for this is not my area expertise... but common mistake I see. Is it properly encoded? Did you add an extra new line (pretty common mistake)? Did you try a curl with the same token? As the docs say you can also use any other standard HTTP setting > **[Configure modules | Metricbeat Reference \[8.17\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-metricbeat.html#module-http-config-options)** So you could try `headers` as well ```auto headers: Authorization: "Bearer mytoken123" ``` ---
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 13, 2025, 3:25am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/9 "2025-02-13T03:25:17Z")
ya tried in post man with same token and working fine. Raised a Q in beats. Thanks for your support. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 13, 2025, 3:26am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/10 "2025-02-13T03:26:15Z")
Try the headers option that should be equivalent of POST man `-H` ---
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 13, 2025, 3:28am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/11 "2025-02-13T03:28:45Z")
> [@stephenb](#): > > ```auto > headers: > Authorization: "Bearer mytoken123" > > ``` Tried and same response.. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 13, 2025, 3:49am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/12 "2025-02-13T03:49:25Z")
Did you base64 encode it? ---
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 13, 2025, 3:52am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/13 "2025-02-13T03:52:30Z")
postman generate in base64 only. In logstash below config working for me: ```auto headers => { "Authorization" => "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ijk0MGRjNzBkLTUyM2MtNDNlOS05YzZkLWI2ZWU3MDMxOWYzMyJ9.eyJzdWIiOiJiZTUwYmZkMS1hOGQ4LTRmZjItOGQ0YS0wNTQzZDY0OWJmMjUiLCJncmFudF90eXBlIjoiY2xpZW50X2NyZWRlbnRpYWxzIiwidG9rZW5fdXNlIjpudWxsLCJzY29wZSI6IkNVU1RPTS1TQ09QRVMiLCJpc3MiOiJFVEYiLCJleHAiOjE3Mzk0MTc5NjIsImlhdCI6MTczOTQxNDM2MiwiY2xpZW50X2lkIjoiYmU1MGJmZDEtYThkOC00ZmYyLThkNGEtMDU0M2Q2NDliZjI1IiwidElkIjoiZTViZjZmYzctNDU3YS00MDZmLWIyZGYtMWEzOGJlOTk4ZjUxIiwianRpIjoiMGQ4NjM4ZTktMzE1OS00YmYzLTg1MGEtZjVhZWM4MTU3NDYwIn0.MzNr7h5JJnNeMJYAKNUIaJn_Vvv2x3qkFxGMTsYh-ZA9d3FXOVN0cHFhKeRu5pxIRPutabDTOz_fFh9HXEbXhqp7ZM8Nmh56VJ8N7qe8lg5eBSsNF5pt3-nxLNS81Qw5Wc50vRVre74KGaOtFpMQjU7BKBnHZtJgii5---paCjw0Bc7uFiPoEoAase6D5nY2bz8Jq3y-7PaBBmoYUAmgraTd8IlVfb07ULSA" } ``` Same token not working in beats. I am saving token file as txt, hope that is OK? Below is the token.txt file content: ```auto "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6Ijk0MGRjNzBkLTUyM2MtNDNlOS05YzZkLWI2ZWU3MDMxOWYzMyJ9.eyJzdWIiOiJiZTUwYmZkMS1hOGQ4LTRmZjItOGQ0YS0wNTQzZDY0OWJmMjUiLCJncmFudF90eXBlIjoiY2xpZW50X2NyZWRlbnRpYWxzIiwidG9rZW5fdXNlIjpudWxsLCJzY29wZSI6IkNVU1RPTS1TQ09QRVMiLCJpc3MiOiJFVEYiLCJleHAiOjE3Mzk0MTc5NjIsImlhdCI6MTczOTQxNDM2MiwiY2xpZW50X2lkIjoiYmU1MGJmZDEtYThkOC00ZmYyLThkNGEtMDU0M2Q2NDliZjI1IiwidElkIjoiZTViZjZmYzctNDU3YS00MDZmLWIyZGYtMWEzOGJlOTk4ZjUxIiwianRpIjoiMGQ4NjM4ZTktMzE1OS00YmYzLTg1MGEtZjVhZWM4MTU3NDYwIn0.MzNr7h5JJnNeMJYAKNUIaJn_Vvv2x3qkFxGMTsYh-ZA9d3FXOVN0cHFhKeRu5pxIRPutabDTOz_fFh9HXEbXhqp7ZM8Nmh56VJ8N7qe8lg5eBSsNF5pt3-nxLNS81Qw5Wc50vRVre74KGaOtFpMQjU7BKBnHZtJgii5-LuwPO0NOOGaUnBnvZ8FCkCvG8M_dCl--paCjw0Bc7uFiPoEoAase6D5nY2bz8Jq3y-7PaBBmoYUAmgraTd8IlVfb07ULSA" ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 13, 2025, 4:04am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/14 "2025-02-13T04:04:56Z")
Don't put the word `Bearer` in the token file just the token... Not sure if quotes or not are needed Hmmm should not be this hard.... ---
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 13, 2025, 4:30am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/15 "2025-02-13T04:30:58Z")
Tried all 3 options, without Bearer, removing quotes, with quotes etc..No luck 🤔 ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 13, 2025, 4:48am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/16 "2025-02-13T04:48:15Z")
So you are hitting the actuator endpoint... Is that what you want ... Not the metrics endpoint? Something Basic going on.. Go back to using the header option... And try the /metrics endpoint Are you using the correct port? Start metricbeat like this `./metricbeat -e -d "*" ` And check the logs output ... There will be a lot See if you get more information Can you `curl` without postman from this same host using curl -H Does it work? Under the covers metricbeat does the same If `curl` works from the same host metricbeat should / will `curl http://endpoint:port/metrics -H "Authorization: Bearer token"` BTW is it http or https? Something basic going on ... ---
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 13, 2025, 4:57am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/17 "2025-02-13T04:57:04Z")
> [@stephenb](#): > > `Bearer ` Below is cur config, so metrics path, i shd change to /metrics? it will not work ```auto metricbeat.modules: - module: prometheus period: 10s hosts: ["http://testserver.dev.com:42115"] metrics_path: /rpsservice/actuator/prometheus?tId=e5bf6fc7-457a-406f-b2df-1a38be998f51 #bearer_token_file: /opt/hc/metricbeat-8.17.2-linux-x86_64/token.txt headers: Authorization: "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjViOWM3N2FiLTY4OTktNDlhNC1hYzA5LWMxYzE2OWNjMGQ0NCJ9.eyJzdWIiOiJiZTUwYmZkMS1hOGQ4LTRmZjItOGQ0YS0wNTQzZDY0OWJmMjUiLCJncmFudF90eXBlIjoiY2xpZW50X2NyZWRlbnRpYWxzIiwidG9rZW5fdXNlIjpudWxsLCJzY29wZSI6IkNVU1RPTS1TQ09QRVMiLCJpc3MiOiJFVEYiLCJleHAiOjE3Mzk0MjYwMTEsImlhdCI6MTczOTQyMjQxMSwiY2xpZW50X2lkIjoiYmU1MGJmZDEtYThkOC00ZmYyLThkNGEtMDU0M2Q2NDliZjI1IiwidElkIjoiZTViZjZmYzctNDU3YS00MDZmLWIyZGYtMWEzOGJlOTk4ZjUxIiwianRpIjoiYWE0MGY5ZjYtYWMxNy00ZjMyLWFlZTAtOTk5ZjA1ZDZlN2NhIn0.NPsNP31m4x6E4FQxaPHVgvGvKlWvE671WBP2txTlUVH4X1NGBAUn7y1Po4YNa63751-Y0MmiducdUak19twB41ltIo062BJVFghN9aRA5GQhwRvOxQlWKeNRXPwb_9exu3mR2KdjPh7uXSqXlC3Uh56fSuTtH_e_ib4ulyUsRMdpvzPHd36dujnEcp5fp4wvGUw7bzCxlNzK4B3OHpViSVUSuBwhdBKAmYHUJoLdZqvllHBi3PQFJzktE-" ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 13, 2025, 4:58am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/18 "2025-02-13T04:58:46Z")
My point is run `curl` from command line and get that to work .. So run `curl` with your endpoint... Get `curl` to work The we will configure metricbeat I am not at your keyboard to debug... Try all options You might need to put your `metric_path` value in quotes since it is not simple `curl "http://testserver.dev.com:42115/disservice/actuator/prometheus?tId=e5bf6fc7-457a-406f-b2df-1a38be998f51" -H "Authorization: Bearer token"` ---
**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\ **Post date:** [February 13, 2025, 5:03am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/19 "2025-02-13T05:03:46Z")
Curl is working just fine with below cmd, able to get all the data ```auto curl http://testserver.dev.com:42115/rpsservice/actuator/prometheus?tId=e5bf6fc7-457a-406f-b2df-1a38be998f51 -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjViOWM3N2FiLTY4OTktNDlhNC1hYzA5LWMxYzE2OWNjMGQ0NCJ9..NPsNP31m4x6E4FQxaPHVgvGvKlWvE671WBP2txTlUVH4X1NGBAUn7y1Po4YNa63751-Y0MmiducdUak19twB41ltIo062BJVFghN9aRA5GQhwRvOxQlWKeNRXPwb_9exu3mR2KdjPh7uXSqXlC3Uh56fSuTtH_e_ib4ulyUsRMdpvzPHd36dujnEcp5fp4wvGUw7bzCxlNzK4B3OHpViSVUSuBwhdBKAmYHUJoLdZqvllHBi3PQFJzktE-wT7fjSztfYYP3XHioJFwKXQfEm8Q1r0yDGDTOLznxLiXw1_PDiBMqoovx3ige133mcQ4_okIyQLNbaFNuTSEMwPPPIHA" ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 13, 2025, 5:07am UTC](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447/20 "2025-02-13T05:07:32Z")
> [@Bhanu\_Praveen](#): > > Curl is working just fine with below cmd, able to get all the data Good Put the `metrics_path` in `"`s [Next page](https://discuss.elastic.co/t/not-able-to-parse-prometheus-metrics-to-json-format/374447.md?page=2)