Need help with splitting a log using logstash split filter

Elastic Stack Logstash
1

Hi,

I receive json logs with the structure as shown below on logstash from a remote server.

{ "timestamp: "...".
  "message": {
       "records": [
             { result ... },
             { result ... },
             { result ... },
       ],
       "records": [
             ......
       ],
      ......
      .......
      "records": [
        .......
      ]
   }
}

Each record is an array of results and there can be a variable number of records in each message.
My requirement is to split this message into a flat structure, such that each new message will have one result in it.

I tried applying the split filter to this message as below:

filter {
split { field => "records" }
}

When I do this, what I observes is that I get multiple messages and each message consists of one result from the first instance of records . However, the new messages still have the other instances (second, third etc.) of records arrays intact.

I am at a loss on how to solve this and would appreciate it if anyone can suggest a solution.

2

You are saying you have multiple values with the same key in a hash? That seems unlikely.

3

I double checked and yes it this that way. This is not a single json document, but is a collection of records of json format that are read from a cloud based kafka like message bus, where multiple records are packed together into a single message.

4

That's not good. If you have an incoming message like

{
    "message": {
        "records": [ { "foo" : 1 }, { "foo" : 2 }, { "foo" : 3 } ],
        "records": [ { "bar" : 1 }, { "bar" : 2 }, { "bar" : 3 } ]
     }
}

and you try to parse that using a json filter or a json codec the second [message][records] field overwrites the first. You will never see the "foo" data.

You could write a custom parser in a ruby filter. Or perhaps you can make it work using a multiline codec to consume a single [message][records] array and then use mutate to adjust it to be valid JSON.

5

sigh!, was hoping for a miracle. Thanks for your comments.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.