# Logstash and JSON array split \[SOLVED\]

**URL:** <https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841>\
**Category:** Logstash\
**Created:** [November 3, 2016, 10:57am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841 "2016-11-03T10:57:33Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Emmanuel\_CHANSON](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emmanuel_chanson/32/12938_2.png) [@Emmanuel\_CHANSON](https://discuss.elastic.co/u/Emmanuel_CHANSON)\
**Post date:** [November 3, 2016, 10:57am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/1 "2016-11-03T10:57:33Z")

</div>

Hello,

I have the following input in JSON file:  
{"jobs":[  
{"name":"Onboarding\_Build","builds":[{"duration":127431,"number":2480,"timestamp":1477608643723},{"duration":132275,"number":2479,"timestamp":1477605046524},{"duration":128311,"number":2478,"timestamp":1477603244388},{"duration":131157,"number":2477,"timestamp":1477601742760},{"duration":130942,"number":2476,"timestamp":1477594241970},{"duration":131681,"number":2475,"timestamp":1477592443221},{"duration":130358,"number":2474,"timestamp":1477591845485},{"duration":134905,"number":2473,"timestamp":1477590943481},{"duration":125540,"number":2472,"timestamp":1477583442267},{"duration":126236,"number":2471,"timestamp":1477581042801}]},

{"name":"Onboarding\_Deploy\_E1DEV4","builds":[{"duration":792391,"number":2255,"timestamp":1477608783769},  
....

The actual config create one event in elasticsearch.

I am looking for creating multiple event based on this array:  
"builds":[{"duration":792391,"number":2255,"timestamp":1477608783769},

How to split the JSON line in order to have one event per build AND use the timestamp value as the @datetime field ?

Thanks by advance.

BR

Emmanuel

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [November 3, 2016, 11:00am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/2 "2016-11-03T11:00:55Z")

</div>

you can use two split filters : [https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html)

first on the "jobs" field to create 1 event per job, then another split on builds to create 1 event per build (per job)

---

<div class="post-metadata">

**Author:** ![Emmanuel\_CHANSON](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emmanuel_chanson/32/12938_2.png) [@Emmanuel\_CHANSON](https://discuss.elastic.co/u/Emmanuel_CHANSON)\
**Post date:** [November 3, 2016, 11:25am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/3 "2016-11-03T11:25:06Z")

</div>

Thanks, I try to use split but no succeed for the moment.  
There is only one job array per JSON file then couple name/build with build that is an array:

{"jobs":[  
{"name":"Onboarding\_Build","builds":[  
{"duration":127431,"number":2480,"timestamp":1477608643723}, {"duration":132275,"number":2479,"timestamp":1477605046524},{"duration":128311,"number":2478,"timestamp":1477603244388},{"duration":131157,"number":2477,"timestamp":1477601742760},{"duration":130942,"number":2476,"timestamp":1477594241970},{"duration":131681,"number":2475,"timestamp":1477592443221},{"duration":130358,"number":2474,"timestamp":1477591845485},{"duration":134905,"number":2473,"timestamp":1477590943481},{"duration":125540,"number":2472,"timestamp":1477583442267},{"duration":126236,"number":2471,"timestamp":1477581042801}  
]},

{"name":"Onboarding\_Deploy\_E1DEV4","builds":[{"duration":792391,"number":2255,"timestamp":1477608783769},{"duration":752825,"number":2254,"timestamp":1477605191572},{"duration":773871,"number":2253,"timestamp":1477603384440},{"duration":748282,"number":2252,"timestamp":1477601885590},{"duration":721893,"number":2251,"timestamp":1477594382020},{"duration":761717,"number":2250,"timestamp":147759

I try to have the following:

{"jobs":[  
{"name":"Onboarding\_Build","builds":[  
{"duration":127431,"number":2480,"timestamp":1477608643723}  
]}]}

{"jobs":[  
{"name":"Onboarding\_Build","builds":[  
{"duration":127431,"number":2480,"timestamp":1477608643723}  
]}]}

or just

{[  
{"name":"Onboarding\_Build","builds":[  
{"duration":127431,"number":2480,"timestamp":1477608643723}  
]}]}  
...

And be able to assign the builds.timestamp value as @datetime

My filter config:

filter {  
#json{  
#source =\> "message"  
#add\_field =\>  
#{  
# "kname" =\> "%{[job.name](http://job.name)}"  
# "kbuilds" =\> "%{job[builds]}"  
#}  
#target =\> "kmsg"  
#}  
split {  
field =\> "builds"  
}  
}

I did not create one event per job because there is only one but maybe I have to even if there is only one cause the the job target presence ?

---

<div class="post-metadata">

**Author:** ![Emmanuel\_CHANSON](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emmanuel_chanson/32/12938_2.png) [@Emmanuel\_CHANSON](https://discuss.elastic.co/u/Emmanuel_CHANSON)\
**Post date:** [November 3, 2016, 7:47pm UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/4 "2016-11-03T19:47:45Z")

</div>

The split parameteris not working. Even when using jobs or builds value.

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [November 4, 2016, 2:45pm UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/5 "2016-11-04T14:45:37Z")

</div>

you need:

```auto
filter {
  split { field => "[jobs]" }
  split { field => "[jobs][builds]" }
}

```

---

<div class="post-metadata">

**Author:** ![Emmanuel\_CHANSON](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emmanuel_chanson/32/12938_2.png) [@Emmanuel\_CHANSON](https://discuss.elastic.co/u/Emmanuel_CHANSON)\
**Post date:** [November 5, 2016, 6:18am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/6 "2016-11-05T06:18:33Z")

</div>

Thanks,

I got this message unfortunately:

....  
"type" =\> "json",  
"tags" =\> [  
[0] "\_split\_type\_failure"  
]  
}

and no data injected

---

<div class="post-metadata">

**Author:** ![Emmanuel\_CHANSON](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emmanuel_chanson/32/12938_2.png) [@Emmanuel\_CHANSON](https://discuss.elastic.co/u/Emmanuel_CHANSON)\
**Post date:** [November 5, 2016, 6:20am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/7 "2016-11-05T06:20:33Z")

</div>

My full config file:

input {  
file {  
path =\> "/tmp/CSV/data.json"  
type =\> "json"  
#codec =\> "json"  
sincedb\_path =\> "/dev/null"  
start\_position =\> "beginning"  
}

}

filter {  
split { field =\> "jobs" }  
split { field =\> "[jobs][builds]" }  
#date {  
# match =\> ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z”]  
# remove\_field =\> "timestamp"  
#}  
}

output {  
elasticsearch {  
action =\> "index"  
hosts =\> "localhost"  
#index =\> "logstash-%{+YYYY.MM.dd}"  
index =\> "json"  
workers =\> 1  
}  
stdout {  
codec =\> rubydebug  
}  
}

---

<div class="post-metadata">

**Author:** ![Emmanuel\_CHANSON](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emmanuel_chanson/32/12938_2.png) [@Emmanuel\_CHANSON](https://discuss.elastic.co/u/Emmanuel_CHANSON)\
**Post date:** [November 5, 2016, 6:32am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/8 "2016-11-05T06:32:49Z")

</div>

I see this in logstash-plain.log

[2016-11-05T07:47:35,539][WARN][logstash.filters.split] Only String and Array types are splittable. field:[jobs] is of type = NilClass  
[2016-11-05T07:47:35,545][WARN][logstash.filters.split] Only String and Array types are splittable. field:[jobs][builds] is of type = NilClass

Emmanuel

---

<div class="post-metadata">

**Author:** ![Emmanuel\_CHANSON](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emmanuel_chanson/32/12938_2.png) [@Emmanuel\_CHANSON](https://discuss.elastic.co/u/Emmanuel_CHANSON)\
**Post date:** [November 5, 2016, 7:10am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/9 "2016-11-05T07:10:02Z")

</div>

Ok now it seems to be better but still no data injected :

input {  
file {  
path =\> "/tmp/CSV/data.json"  
type =\> "json"  
codec =\> "json"  
sincedb\_path =\> "/dev/null"  
start\_position =\> "beginning"  
}

}

filter {  
split { field =\> "[jobs]" }  
split { field =\> "[jobs][builds]" }

}

output {  
elasticsearch {  
action =\> "index"  
hosts =\> "localhost"  
#index =\> "logstash-%{+YYYY.MM.dd}"  
index =\> "index\_json"  
workers =\> 1  
}  
stdout {  
codec =\> rubydebug  
}

Output

Sending Logstash logs to /var/log/logstash/ which is now configured via log4j2.properties.  
{  
"path" =\> "/tmp/CSV/data.json",  
"@timestamp" =\> 2016-11-05T07:25:24.323Z,  
"jobs" =\> {  
"name" =\> "Onboarding\_Build",  
"builds" =\> {  
"duration" =\> 127431,  
"number" =\> 2480,  
"timestamp" =\> 1477608643723  
}  
},  
"@version" =\> "1",  
"host" =\> "[ks3309573.kimsufi.com](http://ks3309573.kimsufi.com)",  
"type" =\> "json"  
}  
{  
"path" =\> "/tmp/CSV/data.json",  
"@timestamp" =\> 2016-11-05T07:25:24.323Z,  
"jobs" =\> {  
"name" =\> "Onboarding\_Build",  
"builds" =\> {  
"duration" =\> 132275,  
"number" =\> 2479,  
"timestamp" =\> 1477605046524  
}  
},  
"@version" =\> "1",  
"host" =\> "[ks3309573.kimsufi.com](http://ks3309573.kimsufi.com)",  
"type" =\> "json"  
}  
{  
"path" =\> "/tmp/CSV/data.json",  
"@timestamp" =\> 2016-11-05T07:25:24.323Z,  
"jobs" =\> {  
"name" =\> "Onboarding\_Build",  
"builds" =\> {  
"duration" =\> 128311,  
"number" =\> 2478,  
"timestamp" =\> 1477603244388  
}  
},  
"@version" =\> "1",  
"host" =\> "[ks3309573.kimsufi.com](http://ks3309573.kimsufi.com)",  
"type" =\> "json"  
}  
{  
"path" =\> "/tmp/CSV/data.json",  
"@timestamp" =\> 2016-11-05T07:25:24.323Z,  
"jobs" =\> {  
"name" =\> "Onboarding\_Build",  
"builds" =\> {  
"duration" =\> 131157,  
"number" =\> 2477,  
"timestamp" =\> 1477601742760  
}  
},  
"@version" =\> "1",  
"host" =\> "[ks3309573.kimsufi.com](http://ks3309573.kimsufi.com)",  
"type" =\> "json"  
}

Emmanuel

---

<div class="post-metadata">

**Author:** ![Emmanuel\_CHANSON](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emmanuel_chanson/32/12938_2.png) [@Emmanuel\_CHANSON](https://discuss.elastic.co/u/Emmanuel_CHANSON)\
**Post date:** [November 5, 2016, 7:35am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/10 "2016-11-05T07:35:34Z")

</div>

Ok I succeed ti make it working using the following config:

input {  
file {  
path =\> "/tmp/CSV/data.json"  
type =\> "json"  
codec =\> "json"  
sincedb\_path =\> "/dev/null"  
start\_position =\> "beginning"  
}

}

filter {  
split { field =\> "[jobs]" }  
split { field =\> "[jobs][builds]" }

}

output {  
elasticsearch {  
action =\> "index"  
hosts =\> "localhost"  
#index =\> "logstash-%{+YYYY.MM.dd}"  
index =\> "json"  
workers =\> 1  
}  
stdout {  
codec =\> rubydebug  
}  
}

Thanks Jose

---

<div class="post-metadata">

**Author:** ![Emmanuel\_CHANSON](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emmanuel_chanson/32/12938_2.png) [@Emmanuel\_CHANSON](https://discuss.elastic.co/u/Emmanuel_CHANSON)\
**Post date:** [November 5, 2016, 9:56am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/11 "2016-11-05T09:56:19Z")

</div>

I got it:

date {  
match =\> ["[jobs][builds][timestamp]", "UNIX\_MS"]  
target =\> "@timestamp"  
remove\_field =\> "timestamp"  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:30am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split-solved/64841/12 "2017-07-06T04:30:56Z")

</div>