HTTP Filter Assistance

[JeremyP]

Hello,

I have a situation where an API I'm trying to leverage which requires multiple API HTTP calls to generate the right document.

I have the following document with multiple "id" in an array which I get from a http_poller input plugin.

{
      "@version" => "1",
    "@timestamp" => 2022-02-19T05:15:38.421Z,
          "tags" => [
        [0] "_httprequestfailure"
    ],
     "resources" => [
        [0] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2474
        },
        [1] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2490
        },
        [2] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2491
        },
        [3] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2493
        },
        [4] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2494
        },
        [5] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2538
        },
        [6] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2539
        },
        [7] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2540
        },
        [8] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2714
        },
        [9] {
            "sources" => [
                [0] "site"
            ],
                 "id" => 2716
        }
    ]
}

Here is my logstash config....

input {
  http_poller {
    urls => {
      console =>
      {
        method => get
        url => "https://console:3780/api/3/tags/7/assets?size=5000"
        user => "logstash"
        password => "pass"
      }
    }
    codec => "json"
    schedule => { every => "10s" }
    truststore => "/etc/logstash/downloaded_truststore.jks"
    truststore_password => "pass"
  }
}

filter {

  mutate { remove_field => "links" }

  http {
    url => "https://console:3780/api/3/assets/%{[resources][id]}"
    verb => "GET"
    user => "logstash"
    password => "Pass"
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

I'm getting the following exception...

[2022-02-19T04:47:37,615][ERROR][logstash.filters.http ][main][9df02eff1b30bd7042a7d90794e36e5b6c5c3bb2e33c3a00b06b6d538de12600] error during HTTP request {:url=>"https://console:3780/api/3/assets/%{[resources][id]}", :body=>nil, :client_error=>"Malformed escape pair at index 58: https://console:3780/api/3/assets/%{[resources][id]}"}

I assume this error is due to the array and likely needs a ruby filter. Each ID should be treated as a separate document as well because each ID is then sent to the HTTP filter to generate it's own JSON document. Maybe a split so that each id is a separate document and then perhaps the HTTP filter will work as intended?

[Badger]

I think you just need to add

split { field => "resources" }

and then that "%{[resources][id]}" sprintf reference should be OK.

[JeremyP]

Thanks Badger.... I was overthinking it. I'm still getting HTTP errors but not as a result of the filter anymore. Much appreciated.