# How to split into multiple events dynamically for a given json? Tried from various question in forums

**URL:** <https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745>\
**Category:** Logstash\
**Created:** [February 10, 2019, 9:54am UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745 "2019-02-10T09:54:04Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\
**Post date:** [February 10, 2019, 9:54am UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/1 "2019-02-10T09:54:04Z")

</div>

Hi All  
These question has been asked multiple times in forum as i see but no definite answer.  
I saw one of the Elastic member [answered](https://discuss.elastic.co/t/nested-json-parsing-in-dynamic-field-names/136211/2) in a nice way, but it is not working for me.

I have a json data in this format coming from a input value

> {"page":1,"per\_page":3,"total":12,"total\_pages":4,"data":[{"id":1,"first\_name":"George","last\_name":"Bluth","avatar":"[https://s3.amazonaws.com/uifaces/faces/twitter/calebogden/128.jpg"},{"id":2,"first\_name":"Janet","last\_name":"Weaver","avatar":"https://s3.amazonaws.com/uifaces/faces/twitter/josephstein/128.jpg"},{"id":3,"first\_name":"Emma","last\_name":"Wong","avatar":"https://s3.amazonaws.com/uifaces/faces/twitter/olegpogodaev/128.jpg](https://s3.amazonaws.com/uifaces/faces/twitter/calebogden/128.jpg%22%7D,%7B%22id%22:2,%22first_name%22:%22Janet%22,%22last_name%22:%22Weaver%22,%22avatar%22:%22https://s3.amazonaws.com/uifaces/faces/twitter/josephstein/128.jpg%22%7D,%7B%22id%22:3,%22first_name%22:%22Emma%22,%22last_name%22:%22Wong%22,%22avatar%22:%22https://s3.amazonaws.com/uifaces/faces/twitter/olegpogodaev/128.jpg)"}]}

Now i am using this in the filter to split into multiple events, but it is not splitting

```
filter {
  json {
    source => "message"
    target => "[@metadata][data]"
  }
  ruby {
    code => "

      string_indexed = event.get('[@metadata][data]')
      items_array = string_indexed.keys.sort_by(&:to_i).map do |key|
        string_indexed[key]
      end
      event.set('[@metadata][items-array]', items_array)
    "
  }
  split {
    field => "[@metadata][items-array]"
    target => "[@metadata][single-item]"
  }
  mutate {
    rename => {
      "[@metadata][single-item][id]" => "id"
      "[@metadata][single-item][first_name]" => "first_name"
      "[@metadata][single-item][last_name]" => "last_name"
    }
  }
}

```

Further in logs i get this error

> Ruby exception occurred: undefined method `keys' for nil:NilClass

The above one is not splitting into multiple events.  
I am looking for a ruby code which create multiple events dynamically on a json datset

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 10, 2019, 1:00pm UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/2 "2019-02-10T13:00:56Z")

</div>

Why not just do this?

```
    json { source => "message" }
    split { field => "data" }

```

undefined method `keys' for nil:NilClass suggests that [@metadata][data] is nil, meaning the json filter did not parse. I suggests removing all the @s from @metadata and seeing what you get in stdout { codec =\> rubydebug }

I find that the split fails. items-array looks like this:

```
  "metadata" => {
    "items-array" => [
        [0] 3,
        [1] 1,
        [2] 12,
        [3] 4,
        [4] [
            [0] {
                        "id" => 1,
                "first_name" => "George",
                    "avatar" => "https://s3.amazonaws.com/uifaces/faces/twitter/calebogden/128.jpg",
                 "last_name" => "Bluth"
            },
            [1] {
                        "id" => 2,
                "first_name" => "Janet",
                    "avatar" => "https://s3.amazonaws.com/uifaces/faces/twitter/josephstein/128.jpg",
                 "last_name" => "Weaver"
            },
            [2] {
                        "id" => 3,
                "first_name" => "Emma",
                    "avatar" => "https://s3.amazonaws.com/uifaces/faces/twitter/olegpogodaev/128.jpg",
                 "last_name" => "Wong"
            }
        ]
    ],

```

and that gets you "exception"=\>"undefined method `empty?' for 3:Fixnum" when it tries to split the first value in the array. That's [fixed](https://github.com/logstash-plugins/logstash-filter-split/issues/13) in the most recent (3.1.7) version of the filter.

Once that is fixed. items-array only has one entry for data, so the split filter has no effect and all the renames fail.

---

<div class="post-metadata">

**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\
**Post date:** [February 10, 2019, 2:53pm UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/3 "2019-02-10T14:53:10Z")

</div>

Hi @Badger

What i am looking for is a universal solution for all JSON data from input. I do not want to use split filter to go over multiple records.  
Split filter is single threaded, when used with high volume data lets say over 100K, it creates clones and then split.  
I want to use ruby to identify all the key/values

on the above example  
I upgraded the filter to 3.1.7  
Removed all @ from metadata, still i get same error

> Ruby exception occurred: undefined method `keys' for nil:NilClass

also i observed,i am not getting any target at all from the json filter.  
Do you have any working example on Ruby Filter with iterate over the keys,so that i can get a flat json and then split in one go

Want to try something similar suggested by @yaauie

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 10, 2019, 3:03pm UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/4 "2019-02-10T15:03:36Z")

</div>

But the transformation that Ry wrote is converting

```auto
{ 
"0": { "foo": "a" },
"1": { "bar": "b" }
}

```

into an array

```auto
[
{ "foo": "a" },
{ "bar": "b" }
]

```

You are starting off with an array. Converting it to a different array is not going to make any difference to how the split filter performs.

---

<div class="post-metadata">

**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\
**Post date:** [February 10, 2019, 3:10pm UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/5 "2019-02-10T15:10:08Z")

</div>

Hi @Badger, is there any other way wherein i can convert 1 event in message of json data into multiple events without using the split filter? ( especially with ruby filter)  
Please let me know

Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 10, 2019, 3:49pm UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/6 "2019-02-10T15:49:31Z")

</div>

You said elsewhere that you have 80,000 bytes of JSON. Unless the objects are tiny, I expect that is around 1,000 entries in the array. You also say it takes 4 hours, which is over 10,000 seconds. So over 10 seconds per event. Looking at the [loop](https://github.com/logstash-plugins/logstash-filter-split/blob/5670ae483d019918a30ddf61fcbc55843fc7e0a0/lib/logstash/filters/split.rb#L84) in the split filter, it is hard to imagine how that could be taking 10 seconds.

So, can try running with "--log.level debug", which will cause it to log a line for each entry in the array

```auto
[DEBUG][logstash.filters.split] Split event {:value=>{"last_name"=>"Bluth", "avatar"=>"https://s3.amazonaws.com/uifaces/faces/twitter/calebogden/128.jpg", "id"=>1, "first_name"=>"George"}, :field=>"[metadata][items-array][0]"}

```

Then review the timestamps on those events. In particular, how much of the time is spent in the input and json parsing, and are the events spit out by split regularly spaced, or are there long gaps in the output.

---

<div class="post-metadata">

**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\
**Post date:** [February 10, 2019, 3:57pm UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/7 "2019-02-10T15:57:29Z")

</div>

The one which i mentioned in this thread,i am experimenting with a small toy dataset.  
Which is this one ( [https://reqres.in/api/users](https://reqres.in/api/users))

Actually i have a bigger dataset which is having almost 15 columns and 80,000 records. From that forum discussion thread it is clear that because of split filter, it goes into serial mode and take 4 hrs.  
In order to avoid split filter, i started using ruby and to see if anything i can do to achieve some result.

My ultimate aim is to get immediate result in output, at least for those events which has done processing in filter and should be available immediately in ouput, rather than waiting in queue to complete first split filter then do other operations in serial mode. Which is not acceptable, Logstash is a streaming tool so it should work somehow.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 10, 2019, 4:53pm UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/9 "2019-02-10T16:53:31Z")

</div>

OK, so if you have a piece of JSON that contains 80,000 records, each of which has 15 columns, then it might be 10 MB. And you are going to create 80,000 copies of that, which involves allocate 800 GB of memory (or possibly 1.6 TB if you have a copy of the message in addition to the parsed data). Oh, and this is Java, so I guess every char is two bytes, so perhaps 3.2 TB of memory to be allocated and GC'd. I'm not surprised it takes a long time.

Replacing the json codec with

```
json {
    source => "message"
    target => "data"
    remove_field => ["message"]
}

```

may avoid a second copy of the 10 MB on each event (it it is present -- I'm not sure what you get from an http\_poller with a codec).

Inside the split filter, it is cloning the event [here](https://github.com/logstash-plugins/logstash-filter-split/blob/5670ae483d019918a30ddf61fcbc55843fc7e0a0/lib/logstash/filters/split.rb#L88), which creates a new 10 or 20 MB object, and the next line replaces that with something that only takes a couple of hundred bytes. That's a really expensive way of doing it. We need something more like the UNIX system call vfork, if you are familiar with that.

Instead of cloning the event, just set event\_split to an empty new event and copy over the fields you need like timestamp, host, version, etc. (still doing event\_split.set(@target, value) etc.). Then yield that.

However, the details of replacing the clone with an empty new event are beyond me. You need someone who understand a little more about events than I do.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 10, 2019, 5:45pm UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/10 "2019-02-10T17:45:30Z")

</div>

The following may work. In a file called splitData.rb put

```auto
def register(params)
    @field = params['field']
    @target = params['target']
end

def filter(event)
    data = event.get(@field)
    event.remove(@field)
    a = []
    data.each { |x|
        e = event.clone
        e.set(@target, x)
        a << e
    }
    a
end

```

Then call it using

```
json {
    source => "message"
    remove_field => ["message"]
}
ruby {
    path => '/home/user/splitData.rb'
    script_params => { field => "data" target => "data" }
}

```

The critical point is to remove the data field before call event.clone.

It occurred to me that the split filter ought to be able to do this optimization (remove source before cloning if it is going to be overwritten). Looking at the code it appears that [this line](https://github.com/logstash-plugins/logstash-filter-split/blob/5670ae483d019918a30ddf61fcbc55843fc7e0a0/lib/logstash/filters/split.rb#L70) may be trying to do this. However I don't know what target refers to (not @target, which is never nil) so I am not sure what it does

---

<div class="post-metadata">

**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\
**Post date:** [February 11, 2019, 7:28am UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/11 "2019-02-11T07:28:29Z")

</div>

Hey Thanks Badger, your solution looks much more promising now. Let me try out this and will let you know the result

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 11, 2019, 7:33am UTC](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/12 "2019-03-11T07:33:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.