# Multiple questions about http\_poller input plugin
**URL:** https://discuss.elastic.co/t/multiple-questions-about-http-poller-input-plugin/355329
**Category:** Logstash
**Created:** [March 13, 2024, 4:43pm UTC](https://discuss.elastic.co/t/multiple-questions-about-http-poller-input-plugin/355329 "2024-03-13T16:43:28Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@nilsen](https://discuss.elastic.co/u/nilsen)
#### Post date: [March 13, 2024, 4:43pm UTC](https://discuss.elastic.co/t/multiple-questions-about-http-poller-input-plugin/355329/1 "2024-03-13T16:43:28Z")
Hello, I decided to merge all of my questions into one thread, instead of making multiple threads, considering they are all http\_poller related.
_Goal: I'm trying to get Logstash to do an http request to our local ActiveMQ to get some queue stats. Using the http\_poller input plugin._
**Invalid URL Error**
Regardless of what I try to do, url-encoded, url-decoded, escaping the special characters, I'm not able make Logstash do a request to the URL below. Anyone know if it is possible to make http\_poller input plugin work with the url below?
```auto
urls {
my_url => 'http://localhost:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%220.0.0.0%22,component=addresses,address=%22pm_inboundNormal%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22pm_inboundNormal%22/countMessages()'
...
}
```
**Reusing method, user, password, headers parameters**
All my requests have mostly the same paramters. Having to repeat `url`, `method`, `user`, `password` and `headers` for every request makes the file very large, considering I'm monitoring around 20 queues. Is there any way or reusing the repeated fields?
_Note: I have now put the url parameters from the question above into the `body` field. And `method` is now `post` instead of `get`_
```auto
http_poller {
urls => {
my_first_queue_count => {
url => "http://localhost:8161/console/jolokia?maxDepth=7&maxCollectionSize=50000&ignoreErrors=true&canonicalNaming=false"
method => post
user => "XXXXX"
password => "XXXXX"
body => '{"mbean": "org.apache.activemq.artemis:address=\"my_first_queue\",broker=\"0.0.0.0\",component=addresses,queue=\"my_first_queue\",routing-type=\"anycast\",subcomponent=queues","type": "exec","operation": "countMessages()"}'
headers => {
Origin => "http://localhost:8161"
"Content-Type" => "application/json"
}
}
my_second_queue_count => {
url => "http://localhost:8161/console/jolokia?maxDepth=7&maxCollectionSize=50000&ignoreErrors=true&canonicalNaming=false"
method => post
user => "XXXXX"
password => "XXXXX"
body => '{"mbean": "org.apache.activemq.artemis:address=\"my_second_queue\",broker=\"0.0.0.0\",component=addresses,queue=\"my_second_queue\",routing-type=\"anycast\",subcomponent=queues","type": "exec","operation": "countMessages()"}'
headers => {
Origin => "http://localhost:8161"
"Content-Type" => "application/json"
}
}
}
request_timeout => 30
schedule => { every => "1m"}
}
```
**Merging multiple requests into one event**
Each request/url creates its own event and document. After renaming the return value to my url name:
`rename => { "value" => "%{[@metadata][input][http_poller][request][name]}"}`
I get the result seen below.
```auto
{
"my_first_queue_count" => 133,
"@timestamp" => 2024-03-13T16:15:59.328905493Z,
"@version" => "1"
},
{
"my_second_queue_count" => 24,
"@timestamp" => 2024-03-13T16:15:59.328905493Z,
"@version" => "1"
}
```
However, I want this to be one event/document as seen below.
```auto
{
"my_first_queue_count" => 133,
"my_second_queue_count" => 24,
"@timestamp" => 2024-03-13T16:15:59.328905493Z,
"@version" => "1"
}
```
I tried to google, but could not figure out a way to achieve this. The only workaround I found was having _dynamic_ name/value pair as seen below. But I feel this will make it harder to work with the data in Kibana dashboards, as I will have to do `breakdown` on `name` all the time. But maybe this is the better solution(less fields in the index, but more documents)? Or is there an easy way of solving this?
```auto
{
"name" => "my_first_queue_count",
"count" => 24,
"@timestamp" => 2024-03-13T16:15:59.328905493Z,
"@version" => "1"
}
```
**add\_field as integer or convert dynamic field**
I think this is impossible considering I find multiple old threads talking about how it is not possible, but they are all old, so I will ask again.
I'm looking of a way to create an integer field from a dynamic parameter as seen in the example below. However, seems like it is not possible to use `add_field` to create an integer, or use a dynamic fieldname with `convert`. Without being able to do this, my logstash.conf files becomes so much larger than I feel is needed. Is this still not possible?
```auto
mutate => { add_field => { "%{[@metadata][input][http_poller][request][name]}" => "[value]" } }
mutate => { convert => { "%{[@metadata][input][http_poller][request][name]}" => "integer" } }
```
* * *
Thanks for any answers.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 13, 2024, 6:07pm UTC](https://discuss.elastic.co/t/multiple-questions-about-http-poller-input-plugin/355329/2 "2024-03-13T18:07:39Z")
> [@nilsen](#):
>
> Merging multiple requests into one event
If you are polling once per minute then you could use aggregate to add all the fields to the map and then use push\_map\_as\_event\_on\_timeout with a short timeout (e.g. 30s). Alternatively, create the document\_id from the timestamp rounded to 30s and do upserts in ES. Both solutions assume that the HTTP requests complete quickly.
---
### Author:  [@nilsen](https://discuss.elastic.co/u/nilsen)
#### Post date: [March 14, 2024, 9:42am UTC](https://discuss.elastic.co/t/multiple-questions-about-http-poller-input-plugin/355329/3 "2024-03-14T09:42:27Z")
If I understand this correctly I would have to create my own ID somehow, most likely based on the timestamp, both with the `aggregate` and `fingerprint` solution? And this would only work as long as I can group all the request within for example 1 whole minute together? It does seem like a nifty workaround, but also very limiting. I will go with the name/value pair solution as of now. Thanks for your suggestion.
Another question related to my other questions. I'm currently in a situation where I feel that using the `exec input plugin` with `curl` would be easier. Sounds insane that should be a better solution. Any counterarguments, anything I'm missing here? _(As curl is able to handle my url without getting Invalid URL Error)._
```auto
exec {
command => 'curl -H "Origin http://localhost:8161" -u "xxxxx:xxxxx" http://localhost:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%220.0.0.0%22,component=addresses,address=%my_queue%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22my_queue%22/countMessages()'
interval => 30
}
```
---
### Author:  [@nilsen](https://discuss.elastic.co/u/nilsen)
#### Post date: [March 14, 2024, 10:24am UTC](https://discuss.elastic.co/t/multiple-questions-about-http-poller-input-plugin/355329/4 "2024-03-14T10:24:06Z")
To answer a few of my own questions after trying some more.
**Invalid URL Error**
I tried again to fiddle with the URL and was able to make it work, unsure what is different now than the hour I tried to do this yesterday.
`http://localhost:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%220.0.0.0%22,component=addresses,address=%22my_queue%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22my_queue%22/countMessages()` is working as excepted now.
**Reusing method, user, password, headers parameters**
I missed that I was able to reuse the `user` and `pass` params.
Because I'm now able to get the url to work I can skip the `body` and `method` params, and move `user` and `password` outside of each url, so the file has become a lot smaller.
```auto
input {
http_poller {
urls => {
my_queue_count => {
url => 'http://localhost:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%220.0.0.0%22,component=addresses,address=%22my_queue%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22my_queue%22/countMessages()'
headers => {
Origin => "http://localhost:8161"
}
}
}
user => "XXXX"
password => "XXXXX"
request_timeout => 30
schedule => { every => "1m"}
}
}
```
Seems like I'm not able to move `headers` outside each URL, so I won't be able to have one-lines for each queue. Maybe I will just have to change ActiveMQ to allow wildcard (\*) `Origin`.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 11, 2024, 10:24am UTC](https://discuss.elastic.co/t/multiple-questions-about-http-poller-input-plugin/355329/5 "2024-04-11T10:24:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.