Hello, I decided to merge all of my questions into one thread, instead of making multiple threads, considering they are all http_poller related.
Goal: I'm trying to get Logstash to do an http request to our local ActiveMQ to get some queue stats. Using the http_poller input plugin.
Invalid URL Error
Regardless of what I try to do, url-encoded, url-decoded, escaping the special characters, I'm not able make Logstash do a request to the URL below. Anyone know if it is possible to make http_poller input plugin work with the url below?
urls {
my_url => 'http://localhost:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%220.0.0.0%22,component=addresses,address=%22pm_inboundNormal%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22pm_inboundNormal%22/countMessages()'
...
}
Reusing method, user, password, headers parameters
All my requests have mostly the same paramters. Having to repeat url
, method
, user
, password
and headers
for every request makes the file very large, considering I'm monitoring around 20 queues. Is there any way or reusing the repeated fields?
Note: I have now put the url parameters from the question above into the body
field. And method
is now post
instead of get
http_poller {
urls => {
my_first_queue_count => {
url => "http://localhost:8161/console/jolokia?maxDepth=7&maxCollectionSize=50000&ignoreErrors=true&canonicalNaming=false"
method => post
user => "XXXXX"
password => "XXXXX"
body => '{"mbean": "org.apache.activemq.artemis:address=\"my_first_queue\",broker=\"0.0.0.0\",component=addresses,queue=\"my_first_queue\",routing-type=\"anycast\",subcomponent=queues","type": "exec","operation": "countMessages()"}'
headers => {
Origin => "http://localhost:8161"
"Content-Type" => "application/json"
}
}
my_second_queue_count => {
url => "http://localhost:8161/console/jolokia?maxDepth=7&maxCollectionSize=50000&ignoreErrors=true&canonicalNaming=false"
method => post
user => "XXXXX"
password => "XXXXX"
body => '{"mbean": "org.apache.activemq.artemis:address=\"my_second_queue\",broker=\"0.0.0.0\",component=addresses,queue=\"my_second_queue\",routing-type=\"anycast\",subcomponent=queues","type": "exec","operation": "countMessages()"}'
headers => {
Origin => "http://localhost:8161"
"Content-Type" => "application/json"
}
}
}
request_timeout => 30
schedule => { every => "1m"}
}
Merging multiple requests into one event
Each request/url creates its own event and document. After renaming the return value to my url name:
rename => { "value" => "%{[@metadata][input][http_poller][request][name]}"}
I get the result seen below.
{
"my_first_queue_count" => 133,
"@timestamp" => 2024-03-13T16:15:59.328905493Z,
"@version" => "1"
},
{
"my_second_queue_count" => 24,
"@timestamp" => 2024-03-13T16:15:59.328905493Z,
"@version" => "1"
}
However, I want this to be one event/document as seen below.
{
"my_first_queue_count" => 133,
"my_second_queue_count" => 24,
"@timestamp" => 2024-03-13T16:15:59.328905493Z,
"@version" => "1"
}
I tried to google, but could not figure out a way to achieve this. The only workaround I found was having dynamic name/value pair as seen below. But I feel this will make it harder to work with the data in Kibana dashboards, as I will have to do breakdown
on name
all the time. But maybe this is the better solution(less fields in the index, but more documents)? Or is there an easy way of solving this?
{
"name" => "my_first_queue_count",
"count" => 24,
"@timestamp" => 2024-03-13T16:15:59.328905493Z,
"@version" => "1"
}
add_field as integer or convert dynamic field
I think this is impossible considering I find multiple old threads talking about how it is not possible, but they are all old, so I will ask again.
I'm looking of a way to create an integer field from a dynamic parameter as seen in the example below. However, seems like it is not possible to use add_field
to create an integer, or use a dynamic fieldname with convert
. Without being able to do this, my logstash.conf files becomes so much larger than I feel is needed. Is this still not possible?
mutate => { add_field => { "%{[@metadata][input][http_poller][request][name]}" => "[value]" } }
mutate => { convert => { "%{[@metadata][input][http_poller][request][name]}" => "integer" } }
Thanks for any answers.