HTTP Filter - Unnest Array

Elastic Stack Logstash
1

Hi there,
I need to unnest an array (based on the workaround mentioned in this previous thread) into seperate documents.

I am calling an API and getting in return an array of results. Each of these results should preferably be seperated into its own document.

Currently all these results are stored inside a body field (see workaround, this seems to be the only option atm ). I assume that you could probably loop over each element in the array and then create a new document for each, with a ruby script. Unfortunanely I never wrote anything Ruby related let alone Ruby + Logstash.
Maybe someone could point me to a good starting point and/or give me an example if possible.

Here is the current output as reference:

{
    "@timestamp" => 2021-07-05T06:25:43.321Z,
        "origin" => "XXX",
      "@version" => "1",
       "headers" => {
                             "date" => "Mon, 05 Jul 2021 06:25:44 GMT",
        "strict-transport-security" => "max-age=15724800; includeSubDomains",
                     "content-type" => "application/json",
                    "cache-control" => "max-age=0, private, must-revalidate",
                             "vary" => "Accept-Encoding",
                     "x-request-id" => "24e8d9f20bf78f24f586980a17d47cef",
                       "connection" => "keep-alive",
                "transfer-encoding" => "chunked",
                         "per-page" => "100",
                        "x-runtime" => "0.023585",
                            "total" => "9",
                             "etag" => "W/\"5f5dd0d16cda9863332b6bac2480dffb\""
    },
        "apiurl" => "XXX&date=2021-07-05",
          "body" => [
        [0] {
            "received_at" => "2021-07-05T07:39:47.000+02:00",
                 "length" => 4,
                 "amount" => 0.118,
             "answer_uri" => nil,
                     "to" => "XXX",
            "description" => "XXX ",
                   "from" => "XXX"
        },
        [1] {
            "received_at" => "2021-07-05T07:41:33.000+02:00",
                 "length" => 587,
                 "amount" => 1.18,
             "answer_uri" => nil,
                     "to" => "XXX",
            "description" => "XXX ",
                   "from" => "XXX"
        },
        [2] {
            "received_at" => "2021-07-05T07:42:38.000+02:00",
                 "length" => 132,
                 "amount" => 0.036,
             "answer_uri" => nil,
                     "to" => "XXX",
            "description" => "XXX",
                   "from" => "XXX"
        }
    ],
           "url" => "https://httpbin.org/get",
          "args" => {},
          "time" => "2021-07-05"
}

Thank you guys so much!
Kind Regards
Marvin

2

You could use a split filter.

3

Ah yes, i completely missed the split filter in the doc. Unfortunately I can't do further actions on the fields inside the body. (It's probably a similiar behaviour to the http-poller problem from my other thread). Therefore this ain't really a solution for my usecase.

I need to access the fields to rename them. At the moment I don't know if this is even possible (with or without the split filter). The "split" documentation didn't mention anything similiar.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.