# Metricbeat http module cant parse query key with period in it

**URL:** <https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540>\
**Category:** Beats\
**Tags:** beats-module, metricbeat\
**Created:** [August 21, 2022, 8:27am UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540 "2022-08-21T08:27:30Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nimar\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nimar_arora/32/109910_2.png) [@Nimar\_Arora](https://discuss.elastic.co/u/Nimar_Arora)\
**Post date:** [August 21, 2022, 8:27am UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/1 "2022-08-21T08:27:30Z")

</div>

`query` key-values such as `resource.kafka.id: XXXX` are being encoded as `resource=map%5Bkafka%3Amap%5Bid%3AXXXX%5D%5D` instead of `resource.kafka.id=XXXX`

This is my metricbeat http module configuration:

```auto
- module: http
  metricsets:
    - json
  period: 1m
  hosts: ["https://api.telemetry.confluent.cloud:443"]
  ssl.verification_mode: "none"
  namespace: "http"
  path: "/v2/metrics/cloud/export"
  query:
    resource.kafka.id: XXXX
  method: "GET"
  username: ...
  password: ...

```

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 22, 2022, 4:29pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/2 "2022-08-22T16:29:16Z")

</div>

Hey @Nimar_Arora,

I think that there is a general issue with configuration keys that contains dots, see [https://github.com/elastic/beats/issues/27079](https://github.com/elastic/beats/issues/27079)

Would using a more json-like syntax work?

```auto
  query: { "resource.kafka.id": XXXX }

```

---

<div class="post-metadata">

**Author:** ![Nimar\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nimar_arora/32/109910_2.png) [@Nimar\_Arora](https://discuss.elastic.co/u/Nimar_Arora)\
**Post date:** [August 22, 2022, 8:31pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/3 "2022-08-22T20:31:23Z")

</div>

Thanks for the link to the existing issue and the suggestion. The JSON-like syntax didn't work, unfortunately. It produced the same problem URL.

I also tried the following:

```auto
  query:
    { 'resource%2Ekafka%2Eid': 'XXXX' }

```

Which resulted in the following query being sent:

```auto
resource%252Ekafka%252Eid=XXXX

```

If there was a way to tell elastic not to encode the query then this last approach might have worked.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 23, 2022, 7:19am UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/4 "2022-08-23T07:19:00Z")

</div>

As this is a GET request, it might work if you put the whole url in `hosts`, did you try something like this?

```auto
- module: http
  metricsets:
    - json
  period: 1m
  hosts: ["https://api.telemetry.confluent.cloud:443/v2/metrics/cloud/export?resource.kafka.id=XXXX"]
  ssl.verification_mode: "none"
  namespace: "http"
  method: "GET"
  username: ...
  password: ...

```

---

<div class="post-metadata">

**Author:** ![Nimar\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nimar_arora/32/109910_2.png) [@Nimar\_Arora](https://discuss.elastic.co/u/Nimar_Arora)\
**Post date:** [August 23, 2022, 1:18pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/5 "2022-08-23T13:18:12Z")

</div>

Yup I tried that as well. This time the error is `invalid character '#' looking for beginning of value`

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 23, 2022, 4:57pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/6 "2022-08-23T16:57:34Z")

</div>

Where there is a `#`? In the `XXXX`?

---

<div class="post-metadata">

**Author:** ![Nimar\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nimar_arora/32/109910_2.png) [@Nimar\_Arora](https://discuss.elastic.co/u/Nimar_Arora)\
**Post date:** [August 23, 2022, 6:29pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/7 "2022-08-23T18:29:44Z")

</div>

There are # in comments in the next block!

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 24, 2022, 9:06am UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/8 "2022-08-24T09:06:12Z")

</div>

Umm, this is weird, could you share the config including these comments? (Obfuscated if needed)

---

<div class="post-metadata">

**Author:** ![Nimar\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nimar_arora/32/109910_2.png) [@Nimar\_Arora](https://discuss.elastic.co/u/Nimar_Arora)\
**Post date:** [August 25, 2022, 9:40am UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/9 "2022-08-25T09:40:18Z")

</div>

OK, I tried this again, here is the full error message:

> metricbeat[19013]: {"log.level":"error","@timestamp":"2022-08-25T09:34:55.013Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset http.json: invalid character '#' looking for beginning of value","service.name":"metricbeat","ecs.version":"1.6.0"}

And here is the full config file `/etc/metricbeat/modules.d/http.yml`

```auto
# Module: http
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/8.3/metricbeat-module-http.html

- module: http
  metricsets:
    - json
  period: 1m
  hosts: ["https://api.telemetry.confluent.cloud:443/v2/metrics/cloud/export?resource.kafka.id=XXX-XXXX"]
  #path: "/v2/metrics/cloud/export"
  #query: {"resource.kafka.id": "XXX-XXXX"}
  ssl.verification_mode: "none"
  namespace: "confluent"
  method: "GET"
  username: YYYY
  password: ZZZZ
  #request.enabled: false
  #response.enabled: false
  #json.is_array: false
  #dedot.enabled: false

- module: http
  #metricsets:
  # - server
  host: "localhost"
  port: "8080"
  enabled: false
  #paths:
  # - path: "/foo"
  # namespace: "foo"
  # fields: # added to the the response in root. overwrites existing fields
  # key: "value"

```

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 25, 2022, 10:52am UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/10 "2022-08-25T10:52:11Z")

</div>

Ah ok, I think this error comes from the parsing of the response received from the server, from [here](https://github.com/elastic/beats/blob/6c1620ad423398b96878f33d40fe49f7fdd0b8c1/metricbeat/module/http/json/json.go#L139), or [here](https://github.com/elastic/beats/blob/6c1620ad423398b96878f33d40fe49f7fdd0b8c1/metricbeat/module/http/json/json.go#L153).

Btw, do you know if the response received is an array? Then `json.is_array: true` should be used.

Would you have a chance to capture this response? Maybe using something like [https://mitmproxy.org/](https://mitmproxy.org/), or modifying Metricbeat to log what is trying to parse.

---

<div class="post-metadata">

**Author:** ![Nimar\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nimar_arora/32/109910_2.png) [@Nimar\_Arora](https://discuss.elastic.co/u/Nimar_Arora)\
**Post date:** [August 25, 2022, 1:21pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/11 "2022-08-25T13:21:35Z")

</div>

Hmm... the response doesn't really look like a JSON object at all. Here is what it looks like

```auto
# HELP confluent_kafka_server_received_bytes The delta count of bytes of the customer's data received from the network. Each sample is the number of bytes received since the previous data sample. The count is sampled every 60 seconds.
# TYPE confluent_kafka_server_received_bytes gauge
confluent_kafka_server_received_bytes{kafka_id="XXXX",topic="XXXX",} N.N NNNNN
confluent_kafka_server_received_bytes{kafka_id="XXXX",topic="XXXXX",} N.N NNNNN

```

So it seems like the problem is that i was incorrect in using the JSON metricset here. You answer above technically solves the issue with the key having a dot in it, so I will mark that as the solution. My problem of getting these metrics into elastic still remains to be solved 🙂

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 25, 2022, 4:40pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/12 "2022-08-25T16:40:50Z")

</div>

Oh, they look like prometheus metrics, try with the [Prometheus module](https://www.elastic.co/guide/en/beats/metricbeat/8.3/metricbeat-module-prometheus.html) then 🙂

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 25, 2022, 4:41pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/13 "2022-08-25T16:41:37Z")

</div>

With the `collector` metricset specifically [Prometheus collector metricset | Metricbeat Reference [8.3] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/8.3/metricbeat-metricset-prometheus-collector.html)

---

<div class="post-metadata">

**Author:** ![Nimar\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nimar_arora/32/109910_2.png) [@Nimar\_Arora](https://discuss.elastic.co/u/Nimar_Arora)\
**Post date:** [August 25, 2022, 5:30pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/14 "2022-08-25T17:30:48Z")

</div>

Wow, yes, that works!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 22, 2022, 7:31pm UTC](https://discuss.elastic.co/t/metricbeat-http-module-cant-parse-query-key-with-period-in-it/312540/15 "2022-09-22T19:31:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.