Multiple questions about http_poller input plugin

[nilsen (Christoffer André Nilsen)]

Hello, I decided to merge all of my questions into one thread, instead of making multiple threads, considering they are all http_poller related.

Goal: I'm trying to get Logstash to do an http request to our local ActiveMQ to get some queue stats. Using the http_poller input plugin.

Invalid URL Error
Regardless of what I try to do, url-encoded, url-decoded, escaping the special characters, I'm not able make Logstash do a request to the URL below. Anyone know if it is possible to make http_poller input plugin work with the url below?

urls {
   my_url => 'http://localhost:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%220.0.0.0%22,component=addresses,address=%22pm_inboundNormal%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22pm_inboundNormal%22/countMessages()' 
...
}

Reusing method, user, password, headers parameters
All my requests have mostly the same paramters. Having to repeat url, method, user, password and headers for every request makes the file very large, considering I'm monitoring around 20 queues. Is there any way or reusing the repeated fields?
Note: I have now put the url parameters from the question above into the body field. And method is now post instead of get

http_poller {
    urls => {
        my_first_queue_count => {
            url => "http://localhost:8161/console/jolokia?maxDepth=7&maxCollectionSize=50000&ignoreErrors=true&canonicalNaming=false"
            method => post
            user => "XXXXX"
            password => "XXXXX"
            body => '{"mbean": "org.apache.activemq.artemis:address=\"my_first_queue\",broker=\"0.0.0.0\",component=addresses,queue=\"my_first_queue\",routing-type=\"anycast\",subcomponent=queues","type": "exec","operation": "countMessages()"}'
            headers => {
                Origin => "http://localhost:8161"
                "Content-Type" => "application/json"
            }
        }
        my_second_queue_count => {
                        url => "http://localhost:8161/console/jolokia?maxDepth=7&maxCollectionSize=50000&ignoreErrors=true&canonicalNaming=false"
                        method => post
                        user => "XXXXX"
                        password => "XXXXX"
                        body => '{"mbean": "org.apache.activemq.artemis:address=\"my_second_queue\",broker=\"0.0.0.0\",component=addresses,queue=\"my_second_queue\",routing-type=\"anycast\",subcomponent=queues","type": "exec","operation": "countMessages()"}'
                        headers => {
                            Origin => "http://localhost:8161"
                            "Content-Type" => "application/json"
            }
        }
    }
    request_timeout => 30
    schedule => { every => "1m"}
}

Merging multiple requests into one event
Each request/url creates its own event and document. After renaming the return value to my url name:
rename => { "value" => "%{[@metadata][input][http_poller][request][name]}"}
I get the result seen below.

{
   "my_first_queue_count" => 133,
   "@timestamp" => 2024-03-13T16:15:59.328905493Z,
   "@version" => "1"
},
{
   "my_second_queue_count" => 24,
   "@timestamp" => 2024-03-13T16:15:59.328905493Z,
   "@version" => "1"
}

However, I want this to be one event/document as seen below.

{
   "my_first_queue_count" => 133,
   "my_second_queue_count" => 24,
   "@timestamp" => 2024-03-13T16:15:59.328905493Z,
   "@version" => "1"
}

I tried to google, but could not figure out a way to achieve this. The only workaround I found was having dynamic name/value pair as seen below. But I feel this will make it harder to work with the data in Kibana dashboards, as I will have to do breakdown on name all the time. But maybe this is the better solution(less fields in the index, but more documents)? Or is there an easy way of solving this?

{
   "name" => "my_first_queue_count",
   "count" => 24,
   "@timestamp" => 2024-03-13T16:15:59.328905493Z,
   "@version" => "1"
}

add_field as integer or convert dynamic field
I think this is impossible considering I find multiple old threads talking about how it is not possible, but they are all old, so I will ask again.

I'm looking of a way to create an integer field from a dynamic parameter as seen in the example below. However, seems like it is not possible to use add_field to create an integer, or use a dynamic fieldname with convert. Without being able to do this, my logstash.conf files becomes so much larger than I feel is needed. Is this still not possible?

mutate => { add_field => { "%{[@metadata][input][http_poller][request][name]}" => "[value]" } }
mutate => { convert => { "%{[@metadata][input][http_poller][request][name]}" => "integer" } }

---

Thanks for any answers.

[Badger]

@nilsen

Merging multiple requests into one event

If you are polling once per minute then you could use aggregate to add all the fields to the map and then use push_map_as_event_on_timeout with a short timeout (e.g. 30s). Alternatively, create the document_id from the timestamp rounded to 30s and do upserts in ES. Both solutions assume that the HTTP requests complete quickly.

[nilsen (Christoffer André Nilsen)]

If I understand this correctly I would have to create my own ID somehow, most likely based on the timestamp, both with the aggregate and fingerprint solution? And this would only work as long as I can group all the request within for example 1 whole minute together? It does seem like a nifty workaround, but also very limiting. I will go with the name/value pair solution as of now. Thanks for your suggestion.

Another question related to my other questions. I'm currently in a situation where I feel that using the exec input plugin with curl would be easier. Sounds insane that should be a better solution. Any counterarguments, anything I'm missing here? (As curl is able to handle my url without getting Invalid URL Error).

exec {
   command => 'curl -H "Origin http://localhost:8161" -u "xxxxx:xxxxx" http://localhost:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%220.0.0.0%22,component=addresses,address=%my_queue%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22my_queue%22/countMessages()'
   interval => 30
}

[nilsen (Christoffer André Nilsen)]

To answer a few of my own questions after trying some more.

Invalid URL Error
I tried again to fiddle with the URL and was able to make it work, unsure what is different now than the hour I tried to do this yesterday.

http://localhost:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%220.0.0.0%22,component=addresses,address=%22my_queue%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22my_queue%22/countMessages() is working as excepted now.

Reusing method, user, password, headers parameters
I missed that I was able to reuse the user and pass params.
Because I'm now able to get the url to work I can skip the body and method params, and move user and password outside of each url, so the file has become a lot smaller.

input {
    http_poller {
        urls => {
            my_queue_count => {
                url => 'http://localhost:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%220.0.0.0%22,component=addresses,address=%22my_queue%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22my_queue%22/countMessages()'
                headers => {
                    Origin => "http://localhost:8161"
                }
            }
        }
        user => "XXXX"
        password => "XXXXX"
        request_timeout => 30
        schedule => { every => "1m"}
    }
}

Seems like I'm not able to move headers outside each URL, so I won't be able to have one-lines for each queue. Maybe I will just have to change ActiveMQ to allow wildcard (*) Origin.