Parsing JSON input with multiple arrays as events

Elastic Stack Logstash
1

Hi,

I am currently using the http_poller input to pull from an API which is giving me 4 arrays which hold 4 different kinds of events. I am trying to move them all into another array and tag each event so that I know which of the 4 event types they are so that they can be sent to the proper indexes and that I can add fields to each based on what fields I expect them to have to make them compatible with the SIEM. I have some ruby code which I thought would loop through each array and add a field to each event but it is just adding the new field to the array instead of each object under the array. Sorry but this is my first shot at ruby and I am not familiar with the syntax.

This is an example of what the api is sending me:

        "@version" : "1",
                     "message" : {
                                "clicksPermitted" : [],
                         "@version" : "1",
                         "@timestamp" : 2020-08-23T23:53:52.551Z,
                         "messagesDelivered" : [],
                                    "clicksBlocked" : [],
                            "messagesBlocked" : []
              }

The messagesBlocked, clicksBlocked, messagesPermitted, and messagesDelivered all hold a number of objects that are the events that I need to extract. Below is the current ruby code that I am trying to use to add a field to each object inside of the messagesBlocked, but the new field is just being added at the end of the messagesBlocked array instead of in each object.

    ruby {
        code => '
          event.get("[message][messagesBlocked]").each {
              event.set("metaEndpoint", "MSGBLK")
          }
        '
      }
2

I would expect that to just keep overwriting the [metaEndpoint] field.

Does a split filter do what you want? (It is not clear to me what you want to do.)

3

The split filter will do what I want if I only have one array of objects to turn into events. What I am trying to do is tag all of the objects under each of the 4 arrays with a new field and then concatenate all 4 arrays into 1 array, remove the original 4 arrays, then split the new combined array into individual events. I can do most of this I just can't figure out how to add the new field to every object in the array. I have just removed the objects in the array because I didn't feel like sanitizing the content. I can give a more complete event if that would help.

4

I would use a ruby filter to merge the arrays

ruby {
    code => 'event.set("mergedArray", event.get("[message][clicksPermitted]") + event.get("[message][messagesDelivered]") + event.get("[message][clicksBlocked]") + event.get("[message][messagesBlocked]"))'
    remove_field => [ "[message][clicksPermitted]", "[message][messagesDelivered]", "[message][clicksBlocked]", "[message][messagesBlocked]" ]
}
split { field => "mergedArray" add_field => { "someField" => "someValue" } }
5

Thanks Badger. That was my plan for merging the arrays but I need to know for each event which array that it came from. I'm going to use that data to add some fields depending on where it came from and then route them to the proper indexes. Since the messagesDelivered and messagesBlocked have the same fields in them and same with the clicks, I won't be able to distinguish them when searching logs and there are significantly different actions that need to be taken depending on if the action was blocked or permitted. Adding that field to each of the array objects before moving them out of the original arrays is where I'm having the issue.

6

If they are arrays of hashes then you could use something like

ruby {
    code => '
        [ "[message][clicksPermitted]", "[message][messagesDelivered]", "[message][clicksBlocked]", "[message][messagesBlocked]" ].each { |arrayName|
            oldArray = event.get(arrayName)
            newArray = []
            oldArray.each { |hash|
                hash["sourceArray"] = arrayName
                newArray << hash
            }
            event.set(arrayName, newArray)
    '
}

I have not tested that, but it should point you in a reasonable direction...

7

Thank you. I'll give that a try. If nothing else it should get me pointed in the right direction.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.