# Json input and splitting
**URL:** https://discuss.elastic.co/t/json-input-and-splitting/232677
**Category:** Logstash
**Created:** [May 14, 2020, 3:23pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677 "2020-05-14T15:23:48Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@tcaetano1995](https://discuss.elastic.co/u/tcaetano1995)
#### Post date: [May 14, 2020, 3:23pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/1 "2020-05-14T15:23:48Z")
Hello, I'm new to the elastic ecosystem.
I'm trying to use logstash to take a json file as input and then use a filter to split into different events but I'm drowned by all of the info there is online.
This is a short version of my code
```auto
{
"proximity": [
{
"network": "all_networks",
"date": "2020-05-12",
"connected": 335,
"visitors": 584,
"passerby": 7201,
"hour": 0
},
{
"network": "all_networks",
"date": "2020-05-12",
"connected": 330,
"visitors": 388,
"passerby": 5829,
"hour": 1
}],
"visitLength": [
{
"network": "all_networks",
"date": "2020-05-12",
"5-20m": 234,
"20-60m": 168,
"1-6h": 99,
"6+h": 83,
"hour": 0
},
{
"network": "all_networks",
"date": "2020-05-12",
"5-20m": 134,
"20-60m": 139,
"1-6h": 115,
"6+h": 0,
"hour": 1
}],
"loyaltyRecords": [
{
"network": "all_networks",
"date": "2020-05-12",
"first": 0,
"daily": 515,
"weekly": 69,
"occasional": 0,
"hour": 0
},
{
"network": "all_networks",
"date": "2020-05-12",
"first": 0,
"daily": 315,
"weekly": 73,
"occasional": 0,
"hour": 1
}]
}
```
Its actually 16k lines but I'm working with this to make faster tests.
I know I have to use an input which I have like this:
```auto
input {
file {
mode => "read"
path => "/home/user/Desktop/data.json"
start_position => "beginning"
codec => json
}
}
```
Filter is
```auto
filter {
json { source => "message"}
split { field => "proximity" }
}
```
and my output
```auto
output {
stdout { }
file { path => "/home/user/Desktop/myfile.txt" codec => json }
}
```
To make things easier I need each of these to be one entry/doc so then I can output to elastic search (for now I'm using the file to check)
```auto
{
"network": "all_networks",
"date": "2020-05-12",
"first": 0,
"daily": 315,
"weekly": 73,
"occasional": 0,
"hour": 1
}
```
As you may have noticed, I have 3 arrays and each contains the "documents" I want to index.
They could be all in the same index or different indices.
Could anyone help me to understand the order of how to do things? i understand there is also a "multiline" code which i don't know if i need or not.
Also im getting a json parse error so im not sure if my data is coded correctly or maybe i need to use the multiline codec for my input.
Thanks in advance
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 14, 2020, 5:06pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/2 "2020-05-14T17:06:52Z")
If you have a working json codec then typically you do not need a json filter as well. However, unless you have complete JSON objects on each line a json codec will not work, you need a multiline codec.
If you want to consume the entire file as a single event then configure the multiline codec as described [here](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/2). Then use a json filter. Then you can split the arrays using filters like
```
split { field => "proximity" }
```
However, that will end up with events that have a proximity field, a visitLength field and a loyaltyRecords field. If that is not what you want then you will need to use ruby.
---
### Author:  [@tcaetano1995](https://discuss.elastic.co/u/tcaetano1995)
#### Post date: [May 14, 2020, 6:29pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/3 "2020-05-14T18:29:53Z")
Thanks for replying so fast.
So I already have used the codec and have the whole json inside the "message" field and called it using the json filter.
Unfortunately the split filter is not doing what i need (as you mentioned previously).
Is there a good guide/link that explains how to get "events" out of each array?
Thanks
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 14, 2020, 9:27pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/4 "2020-05-14T21:27:31Z")
> [@tcaetano1995](#):
>
> Unfortunately the split filter is not doing what i need (as you mentioned previously).
What problem do you have with the results of the split filter?
---
### Author:  [@tcaetano1995](https://discuss.elastic.co/u/tcaetano1995)
#### Post date: [May 15, 2020, 2:31pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/5 "2020-05-15T14:31:08Z")
I have applied the following filters
```
json { source => "message" remove_field => ["message"] }
mutate { remove_field => ["host", "@version", "@timestamp", "path"] }
```
And my output looks like this:
```
{
"visitLength": [(here I have the 2 elements)],
"loyaltyRecords": [(another 2)],
"proximity": [(The last 2)]
}
```
now what I need, is to get each element of all of the arrays out so later I can identify each of them as an event.
If I use the split I get the elements but inside their original field For example
`split { field => "proximity" }`
this gets me this
In your previous comment, you already established that behaviour.
---
### Author:  [@tcaetano1995](https://discuss.elastic.co/u/tcaetano1995)
#### Post date: [May 15, 2020, 3:08pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/6 "2020-05-15T15:08:58Z")
UPDATE
I have found that in ruby I could create a new field with other fields data,
` ruby { code => 'event.set("my_field", [event.get("proximity"), event.get("loyaltyRecords"), event.get("visitLength")]) '}`
Problem is that I keep getting them nested, is there a way to get the all the elements? unless i need to do a "for"???
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 15, 2020, 3:50pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/7 "2020-05-15T15:50:19Z")
If you want to move nested fields to the top level you can do something like [this](https://discuss.elastic.co/t/how-to-dynamically-move-nested-key-value-to-root-level/180006/2).
---
### Author:  [@tcaetano1995](https://discuss.elastic.co/u/tcaetano1995)
#### Post date: [May 15, 2020, 4:14pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/8 "2020-05-15T16:14:26Z")
I replaced the "doc" for one of my fields to get the objects out of the field into top level
That seems to be the solution from what I can read of the code but I'm getting an exception
`Ruby exception occurred: no implicit conversion of Hash into String`
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 15, 2020, 4:33pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/9 "2020-05-15T16:33:33Z")
> [@tcaetano1995](#):
>
> Ruby exception occurred: no implicit conversion of Hash into String
That suggests that you are trying to do it to an array of hashes. You would need to split them first.
What does an event look like if you send it to this?...
```
output { stdout { codec => rubydebug } }
```
---
### Author:  [@tcaetano1995](https://discuss.elastic.co/u/tcaetano1995)
#### Post date: [May 15, 2020, 4:40pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/10 "2020-05-15T16:40:48Z")
Without the split
```
2020-05-15T13:38:34,375][ERROR][logstash.filters.ruby][main] Ruby exception occurred: no implicit conversion of Hash into String
.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"visitLength" => [
[0] {
"hour" => 0,
"1-6h" => 99,
"network" => "all_networks",
"20-60m" => 168,
"date" => "2020-05-12",
"5-20m" => 234,
"6+h" => 83
},
[1] {
"hour" => 1,
"1-6h" => 115,
"network" => "all_networks",
"20-60m" => 139,
"date" => "2020-05-12",
"5-20m" => 134,
"6+h" => 0
}
],
"loyaltyRecords" => [
[0] {
"hour" => 0,
"occasional" => 0,
"network" => "all_networks",
"daily" => 515,
"date" => "2020-05-12",
"first" => 0,
"weekly" => 69
},
[1] {
"hour" => 1,
"occasional" => 0,
"network" => "all_networks",
"daily" => 315,
"date" => "2020-05-12",
"first" => 0,
"weekly" => 73
}
],
"tags" => [
[0] "_rubyexception"
],
"proximity" => [
[0] {
"passerby" => 7201,
"hour" => 0,
"network" => "all_networks",
"date" => "2020-05-12",
"connected" => 335,
"visitors" => 584
},
[1] {
"passerby" => 5829,
"hour" => 1,
"network" => "all_networks",
"date" => "2020-05-12",
"connected" => 330,
"visitors" => 388
}
]
}
```
---
### Author:  [@tcaetano1995](https://discuss.elastic.co/u/tcaetano1995)
#### Post date: [May 15, 2020, 4:46pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/11 "2020-05-15T16:46:27Z")
> [@tcaetano1995](#):
>
> ruby { code =\> 'event.set("my\_field", [event.get("proximity"), event.get("loyaltyRecords"), event.get("visitLength")]) '}
My filters are:
```
json { source => "message" remove_field => ["message"] }
mutate { remove_field => ["host", "@version", "@timestamp", "path"] }
ruby {
code => '
event.get("proximity").each { |k, v|
event.set(k,v)
}
event.remove("proximity")
'
}
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 15, 2020, 9:53pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/12 "2020-05-15T21:53:30Z")
For the event you posted, if you do
```
split { field => "loyaltyRecords" }
split { field => "proximity" }
split { field => "visitLength" }
ruby {
code => '
["loyaltyRecords", "proximity", "visitLength"].each { |field|
event.get(field).each { |k, v|
event.set(k, v)
}
event.remove(field)
}
'
}
```
then you will get events like
```
{
"6+h" => 0,
"network" => "all_networks",
"weekly" => 73,
"visitors" => 388,
"20-60m" => 139,
"@timestamp" => 2020-05-15T21:51:47.643Z,
"date" => "2020-05-12",
"daily" => 315,
"1-6h" => 115,
"connected" => 330,
"passerby" => 5829,
"hour" => 1,
"occasional" => 0,
"5-20m" => 134,
"@version" => "1",
"first" => 0
}
```
Does that work for you?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 12, 2020, 9:53pm UTC](https://discuss.elastic.co/t/json-input-and-splitting/232677/13 "2020-06-12T21:53:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.