Remove json array elements that are objects

[Jason_Brooks]

Hello, I have the following (shortened) json array:

"identifiers": [{ "name": "ENCODED_RECIPIENT_ID", "value": "LTMxNjcxMTc4NjQS1", "isOriginal": false }, { "name": "ENCODED_RECIPIENT_ID", "value": "LTMxNjcxODA5NzQS1", "isOriginal": true }, { "name": "contactId", "value": "-3167117864", "isOriginal": false }, { "name": "contactId", "value": "-3167180974", "isOriginal": true }]

note each array element is its own object. I need to remove all of the objects in this array with "isOriginal": false.

I have played with the following:

split {
field => "identifiers"
}
if ![identifiers][isOriginal] {drop{}}

This creates several events, each with one identifier object with "isOriginal": true.

now, I would like to put the events back together again and have an array of identifiers again. it appears I should use aggregate.

OR, is there a better way to just remove objects from the original array where IsOriginal is false?

Thank you in advance for your help.

--jason

[guyboertje]

Assumed - you have decoding from JSON. The value of the [identifiers] is an Array with a variable number of elements.

You will need a ruby filter for this and I have the code.

Before I post the code, answer this:
** Are you using LS 6.0 or greater? **

[Jason_Brooks]

Hello,

and thank you for your reply!

I am presently using version 6.1.

I had been wondering if I needed to build an external script.

Again, thank you!

--jason

[guyboertje]

With LS 6.1 you have two styles of creating a ruby filter.

1. The old style using code => '...'
2. New style using an external file that is more powerful and can include tests!

I'm guessing that you want the old style.

input {
  generator {
    message => '{"identifiers": [{ "name": "ENCODED_RECIPIENT_ID", "value": "LTMxNjcxMTc4NjQS1", "isOriginal": false }, { "name": "ENCODED_RECIPIENT_ID", "value": "LTMxNjcxODA5NzQS1", "isOriginal": true }, { "name": "contactId", "value": "-3167117864", "isOriginal": false }, { "name": "contactId", "value": "-3167180974", "isOriginal": true }]}'
    count => 1
  }
}

filter {
  json {
    source => "message"
  }
  if "_jsonparsefailure" not in [tags]  {
    ruby {
      code => '
        elements = event.get("[identifiers]")
        if elements.is_a?(Array)
          event.set("[identifiers]", elements.reject{|ele| !ele["isOriginal"]})
        end
      '
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Result:

{
    "identifiers" => [
        [0] {
                 "value" => "LTMxNjcxODA5NzQS1",
            "isOriginal" => true,
                  "name" => "ENCODED_RECIPIENT_ID"
        },
        [1] {
                 "value" => "-3167180974",
            "isOriginal" => true,
                  "name" => "contactId"
        }
    ],
       "@version" => "1",
           "host" => "Elastics-MacBook-Pro.local",
        "message" => "{\"identifiers\": [{ \"name\": \"ENCODED_RECIPIENT_ID\", \"value\": \"LTMxNjcxMTc4NjQS1\", \"isOriginal\": false }, { \"name\": \"ENCODED_RECIPIENT_ID\", \"value\": \"LTMxNjcxODA5NzQS1\", \"isOriginal\": true }, { \"name\": \"contactId\", \"value\": \"-3167117864\", \"isOriginal\": false }, { \"name\": \"contactId\", \"value\": \"-3167180974\", \"isOriginal\": true }]}",
     "@timestamp" => 2018-01-11T17:09:01.267Z,
       "sequence" => 0
}

[Jason_Brooks]

Hello guyboertje,

Thank you! that is not only exactly what I needed but also a push in a direction I knew I was heading!

I would like to clarify a couple things about the logstash environment. Am I correct in assuming that per event, it's pretty much one pass through the filters, and if I want to loop on something, I need to use ruby?

Thanks again!

--jason

[guyboertje]

Yes. More or less.

Some filters allow for multiple attempts at a transformation like grok, date and dissect.
The mutate filter has limited support for transforming array values and it also support multiple actions in any one mutate config declaration, many people don't know this and declare multiple mutate filter blocks with one transform in each - these can be combined into one mutate filter block providing one keeps in mind the order of operations.

We don't have a filter that does general purpose transforms on array values like select or reject. Its quite rare to need these transforms - and we know it can be done with a ruby filter.