Hello,
I have a situation where an API I'm trying to leverage which requires multiple API HTTP calls to generate the right document.
I have the following document with multiple "id" in an array which I get from a http_poller input plugin.
{
"@version" => "1",
"@timestamp" => 2022-02-19T05:15:38.421Z,
"tags" => [
[0] "_httprequestfailure"
],
"resources" => [
[0] {
"sources" => [
[0] "site"
],
"id" => 2474
},
[1] {
"sources" => [
[0] "site"
],
"id" => 2490
},
[2] {
"sources" => [
[0] "site"
],
"id" => 2491
},
[3] {
"sources" => [
[0] "site"
],
"id" => 2493
},
[4] {
"sources" => [
[0] "site"
],
"id" => 2494
},
[5] {
"sources" => [
[0] "site"
],
"id" => 2538
},
[6] {
"sources" => [
[0] "site"
],
"id" => 2539
},
[7] {
"sources" => [
[0] "site"
],
"id" => 2540
},
[8] {
"sources" => [
[0] "site"
],
"id" => 2714
},
[9] {
"sources" => [
[0] "site"
],
"id" => 2716
}
]
}
Here is my logstash config....
input {
http_poller {
urls => {
console =>
{
method => get
url => "https://console:3780/api/3/tags/7/assets?size=5000"
user => "logstash"
password => "pass"
}
}
codec => "json"
schedule => { every => "10s" }
truststore => "/etc/logstash/downloaded_truststore.jks"
truststore_password => "pass"
}
}
filter {
mutate { remove_field => "links" }
http {
url => "https://console:3780/api/3/assets/%{[resources][id]}"
verb => "GET"
user => "logstash"
password => "Pass"
}
}
output {
stdout {
codec => rubydebug
}
}
I'm getting the following exception...
[2022-02-19T04:47:37,615][ERROR][logstash.filters.http ][main][9df02eff1b30bd7042a7d90794e36e5b6c5c3bb2e33c3a00b06b6d538de12600] error during HTTP request {:url=>"https://console:3780/api/3/assets/%{[resources][id]}", :body=>nil, :client_error=>"Malformed escape pair at index 58: https://console:3780/api/3/assets/%{[resources][id]}"}
I assume this error is due to the array and likely needs a ruby filter. Each ID should be treated as a separate document as well because each ID is then sent to the HTTP filter to generate it's own JSON document. Maybe a split so that each id is a separate document and then perhaps the HTTP filter will work as intended?