Split array with multiple identical keys

Elastic Stack Logstash
1

I am working on splitting out an array that has multiple identical values. The message is json, and I run the whole message through a json filter plugin before trying to parse out the "threatsInfoMap" field. Currently it is only capturing the first object in the array, but I would like to capture all of them (my sample array has two objects, I have seen up to 5).

My configuration:

json {
  source => "message"
}   split {
  field => "threatsInfoMap"
}

Sample Data:

"threatsInfoMap":  [
                       {
                           "threatID":  "legitimate_id_code1",
                           "threatStatus":  "active",
                           "classification":  "spam",
                           "threatUrl":  "https://sampleurl.com/1",
                           "threatTime":  "2019-03-31T19:22:46.000Z",
                           "threat":  "valid_ioc1",
                           "campaignID":  "valid_id1",
                           "threatType":  "url"
                       },
                       {
                           "threatID":  "legitimate_id_code2",
                           "threatStatus":  "active",
                           "classification":  "spam",
                           "threatUrl":  "https://sampleurl.com/2",
                           "threatTime":  "2019-03-31T09:39:00.000Z",
                           "threat":  "valid_ioc2",
                           "campaignID":  "valid_id2",
                           "threatType":  "url"
                       }
                   ],
2

I do not understand the question. When you use a split filter each event will contain a single entry from the array. I would expect you to get

{
"threatsInfoMap" => {
          "threatID" => "legitimate_id_code1",
        "threatType" => "url",
        "campaignID" => "valid_id1",
        "threatTime" => "2019-03-31T19:22:46.000Z",
            "threat" => "valid_ioc1",
    "classification" => "spam",
         "threatUrl" => "https://sampleurl.com/1",
      "threatStatus" => "active"
    }
}
{
"threatsInfoMap" => {
          "threatID" => "legitimate_id_code2",
        "threatType" => "url",
        "campaignID" => "valid_id2",
        "threatTime" => "2019-03-31T09:39:00.000Z",
            "threat" => "valid_ioc2",
    "classification" => "spam",
         "threatUrl" => "https://sampleurl.com/2",
      "threatStatus" => "active"
    }
}

as two separate events. What are you getting?

3

I am just getting

{
"threatsInfoMap" => {
      "threatID" => "legitimate_id_code1",
    "threatType" => "url",
    "campaignID" => "valid_id1",
    "threatTime" => "2019-03-31T19:22:46.000Z",
        "threat" => "valid_ioc1",
"classification" => "spam",
     "threatUrl" => "https://sampleurl.com/1",
  "threatStatus" => "active"
 }
}

The second object isn't there.

4

I cannot think of an explanation for that.

5

Thanks for taking a look.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.