# Need help with splitting a log using logstash split filter

**URL:** <https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906>\
**Category:** Logstash\
**Created:** [June 14, 2021, 10:44pm UTC](https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906 "2021-06-14T22:44:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kajira](https://avatars.discourse-cdn.com/v4/letter/k/ad7895/32.png) [@kajira](https://discuss.elastic.co/u/kajira)\
**Post date:** [June 14, 2021, 10:44pm UTC](https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906/1 "2021-06-14T22:44:03Z")

</div>

Hi,

I receive json logs with the structure as shown below on logstash from a remote server.

```auto
{ "timestamp: "...".
  "message": {
       "records": [
             { result ... },
             { result ... },
             { result ... },
       ],
       "records": [
             ......
       ],
      ......
      .......
      "records": [
        .......
      ]
   }
}

```

Each record is an array of results and there can be a variable number of records in each message.  
My requirement is to split this message into a flat structure, such that each new message will have one result in it.

I tried applying the split filter to this message as below:

filter {  
split { field =\> "records" }  
}

When I do this, what I observes is that I get multiple messages and each message consists of one result from the first instance of records . However, the new messages still have the other instances (second, third etc.) of records arrays intact.

I am at a loss on how to solve this and would appreciate it if anyone can suggest a solution.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 14, 2021, 11:31pm UTC](https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906/2 "2021-06-14T23:31:25Z")

</div>

> [@kajira](#):
>
> ```auto
> "message": {
> "records": [
> { result ... },
> { result ... },
> { result ... },
> ],
> "records": [
> ......
> ],
> 
> ```

You are saying you have multiple values with the same key in a hash? That seems unlikely.

---

<div class="post-metadata">

**Author:** ![kajira](https://avatars.discourse-cdn.com/v4/letter/k/ad7895/32.png) [@kajira](https://discuss.elastic.co/u/kajira)\
**Post date:** [June 14, 2021, 11:53pm UTC](https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906/3 "2021-06-14T23:53:15Z")

</div>

I double checked and yes it this that way. This is not a single json document, but is a collection of records of json format that are read from a cloud based kafka like message bus, where multiple records are packed together into a single message.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 15, 2021, 12:18am UTC](https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906/4 "2021-06-15T00:18:00Z")

</div>

> [@kajira](#):
>
> ```auto
> "message": {
> "records": [
> { result ... },
> { result ... },
> { result ... },
> ],
> "records": [
> ......
> ],
> 
> ```

That's not good. If you have an incoming message like

```auto
{
    "message": {
        "records": [{ "foo" : 1 }, { "foo" : 2 }, { "foo" : 3 }],
        "records": [{ "bar" : 1 }, { "bar" : 2 }, { "bar" : 3 }]
     }
}

```

and you try to parse that using a json filter or a json codec the second [message][records] field overwrites the first. You will never see the "foo" data.

You could write a custom parser in a ruby filter. Or perhaps you can make it work using a multiline codec to consume a single [message][records] array and then use mutate to adjust it to be valid JSON.

---

<div class="post-metadata">

**Author:** ![kajira](https://avatars.discourse-cdn.com/v4/letter/k/ad7895/32.png) [@kajira](https://discuss.elastic.co/u/kajira)\
**Post date:** [June 15, 2021, 12:49am UTC](https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906/5 "2021-06-15T00:49:50Z")

</div>

sigh!, was hoping for a miracle. Thanks for your comments.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 13, 2021, 12:50am UTC](https://discuss.elastic.co/t/need-help-with-splitting-a-log-using-logstash-split-filter/275906/6 "2021-07-13T00:50:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.