Split json object into multiple Events for ES

Elastic Stack Logstash
1

Hi,

I need to theorise the below as it will take a while to complete through trial and error due to the speed at which these logs are sent from AWS.

Is it possible to split this json object into multiple events using the split filter - and how do I accomplish this? Something like the below:

filter {
  if [type] =~ /aws_cloudtrail/ {
      json {
        source => "message"
        target => "event_log"
       }

split {
field => "[event_log][Records]"
     }
  }
}

Each time we see the eventID field I want it to be a new event that is parsed into logstash?

"event_log":  {
     "Records": [
      {
      "eventID": "d5e2af26-a54f-49b3-9389-93b5a5fff7b3",
      "awsRegion": "eu-west-2",
      "eventVersion": "1.05",
      "responseElements": null,
      "sourceIPAddress": "apigateway.amazonaws.com",
      "requestParameters": {
        "logGroupName": "API-Gateway-Execution-Logs_a5tmfqthd6/v1",
        "logStreamName": "xxx"
      },
    {
      "eventID": "ed1e96c2-6ea4-4aa4-9d5a-ecc92efcf372",
      "awsRegion": "eu-west-2",
      "eventVersion": "1.05",
      "responseElements": null,
      "sourceIPAddress": "apigateway.amazonaws.com",
      "requestParameters": {
        "logGroupName": "API-Gateway-Execution-Logs_a5tmfqthd6/v1",
        "logStreamName": "6d3c820e60869c8892d7caa4b72824bf"
      }
   ]
 }
2

I would expect that to work.

3

@Badger I kept editing this, and realised eventually I came up with the solution.. Thanks

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.