# Split nested json array

**URL:** <https://discuss.elastic.co/t/split-nested-json-array/147969>\
**Category:** Logstash\
**Created:** [September 10, 2018, 1:32pm UTC](https://discuss.elastic.co/t/split-nested-json-array/147969 "2018-09-10T13:32:27Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![slavo3](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@slavo3](https://discuss.elastic.co/u/slavo3)\
**Post date:** [September 10, 2018, 1:32pm UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/1 "2018-09-10T13:32:27Z")

</div>

I would like to to retrieve every element in below JSON to be a field so as to visualize in kibana by applying metrics in dashboard.

{  
"message":  
"{"SegmentId":4,"VectorList":[{"LocalizationId":60,"ServiceId":1,"Date":"2018-09-10T00:00:00","IsAvailable":true,"FreeTermCount":80,"SegmentFreeTermCount":80,"FilterValue":null},{"LocalizationId":60,"ServiceId":2,"Date":"2018-09-10T00:00:00","IsAvailable":true,"FreeTermCount":21,"SegmentFreeTermCount":21,"FilterValue":42},{"LocalizationId":60,"ServiceId":1,"Date":"2018-09-11T00:00:00","IsAvailable":true,"FreeTermCount":114,"SegmentFreeTermCount":114,"FilterValue":null},{"LocalizationId":60,"ServiceId":2,"Date":"2018-09-11T00:00:00","IsAvailable":true,"FreeTermCount":104,"SegmentFreeTermCount":78,"FilterValue":null}]}"  
}

I need each object in the array VectorList to be a separate entry in Elasticsearch and every attribute like LocalizationId etc to be a field. How would I create filter in configuring Logstash to do this?

I know that i sould use split filter, but my attempts failed. Is there anyone who can help me define the filter?

I can add that message value was created by wcf service using [Json.NET - Newtonsoft](https://www.newtonsoft.com/) so characters like " are generated automatically.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 10, 2018, 1:41pm UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/2 "2018-09-10T13:41:39Z")

</div>

The general idea is to use a json filter to parse the JSON string in the message field and then use a split filter to split the array in the `VectorList` field we got from the json filter.

If you show us what you have so far it'll be easier to help.

---

<div class="post-metadata">

**Author:** ![slavo3](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@slavo3](https://discuss.elastic.co/u/slavo3)\
**Post date:** [September 10, 2018, 2:07pm UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/3 "2018-09-10T14:07:08Z")

</div>

I would like to have output to elasticsearch

{  
"SegmentId":4,  
"LocalizationId":60,  
"ServiceId":1,  
"Date":"2018-09-10T00:00:00",  
"IsAvailable":true,  
"FreeTermCount":80,  
"SegmentFreeTermCount":80,  
"FilterValue":null  
}

{  
"LocalizationId":60,  
"ServiceId":2,  
"Date":"2018-09-10T00:00:00",  
"IsAvailable":true,  
"FreeTermCount":21,  
"SegmentFreeTermCount":21,  
"FilterValue":42  
}  
....

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 10, 2018, 2:11pm UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/4 "2018-09-10T14:11:17Z")

</div>

Yes, I understand that. I'm interested in what your Logstash configuration looks like so far.

---

<div class="post-metadata">

**Author:** ![slavo3](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@slavo3](https://discuss.elastic.co/u/slavo3)\
**Post date:** [September 10, 2018, 2:13pm UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/5 "2018-09-10T14:13:25Z")

</div>

filter {

```
    split { 
        field => "message"
    }
	
	 mutate {
		add_field => { 
			"LocalizationId" => "%{[message][VectorList][ServiceVariantId]"
			"ServiceVariantId" => "%{[message][VectorList][ServiceVariantId]}"
			"Date" => "%{[message][VectorList][Date]}"
			"IsAvailable" => "%{[message][VectorList][IsAvailable]}"
			"FreeTermCount" => "%{[message][VectorList][FreeTermCount]}"
			"SegmentFreeTermCount" => "%{[message][VectorList][SegmentFreeTermCount]}"
			"FilterValue" => "%{[message][VectorList][FilterValue]}"
}

remove_field => ["[message]" ]

```

}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 11, 2018, 5:55am UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/6 "2018-09-11T05:55:08Z")

</div>

Insert a `json { source => "message" }` filter prior to your split filter and change the split filter's `field` option to "VectorList".

---

<div class="post-metadata">

**Author:** ![slavo3](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@slavo3](https://discuss.elastic.co/u/slavo3)\
**Post date:** [September 11, 2018, 8:46am UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/7 "2018-09-11T08:46:01Z")

</div>

Thanks for advice,

my single row in file:  
{"message":"{SegmentId:3,VectorList:[{LocalizationId:60,ServiceId:2,Date:2018-09-11T00:00:00,IsAvailable:true,FreeTermCount:92,SegmentFreeTermCount:92,FilterValue:null},{LocalizationId:60,ServiceVariantId:2,Date:2018-09-12T00:00:00,IsAvailable:true,FreeTermCount:48,SegmentFreeTermCount:34,FilterValue:68},]}"}

I would like you to pay attention to the format  
{"message":"{ ...}"

and when i added  
filter {  
json { source =\> "message" }  
...  
}  
i received error parsing json:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7c102ae187cd0b45f2b3e9855d5d162359b79ade.png)

How can i resolve it?

---

<div class="post-metadata">

**Author:** ![slavo3](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@slavo3](https://discuss.elastic.co/u/slavo3)\
**Post date:** [September 11, 2018, 9:13am UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/8 "2018-09-11T09:13:13Z")

</div>

OK, i repaired my single row and now parsing is ok

{"message":"{"SegmentId":3,"VectorList":[{"LocalizationId":60,"ServiceId":2,"Date":"2018-09-11T00:00:00","IsAvailable":true,"FreeTermCount":86,"SegmentFreeTermCount":86,"FilterValue":null},{"LocalizationId":60,"ServiceId":2,"Date":"2018-09-12T00:00:00","IsAvailable":true,"FreeTermCount":48,"SegmentFreeTermCount":34,"FilterValue":68}]}"}

---

<div class="post-metadata">

**Author:** ![slavo3](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@slavo3](https://discuss.elastic.co/u/slavo3)\
**Post date:** [September 11, 2018, 9:27am UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/9 "2018-09-11T09:27:36Z")

</div>

[Magnus Bäck] Thanks for your help. Everythink is ok, below is my logstash configuration which is correct and generate event per element in my vectorList.

input {  
file {  
type =\> "MyLog"  
path =\> ["C:/logs/X/\*.json"]  
start\_position =\> "beginning"  
codec =\> json  
}   
}

filter {  
json { source =\> "message" }

```
split { 
    field => "VectorList"
}

mutate {
	add_field => { 
		"LocalizationId" => "%{[VectorList][ServiceVariantId]}"
		"ServiceVariantId" => "%{[VectorList][ServiceVariantId]}"
		"Date" => "%{[VectorList][Date]}"
		"IsAvailable" => "%{[VectorList][IsAvailable]}"
		"FreeTermCount" => "%{[VectorList][FreeTermCount]}"
		"SegmentFreeTermCount" => "%{[VectorList][SegmentFreeTermCount]}"
		"FilterValue" => "%{[VectorList][FilterValue]}"
	}
	remove_field => ["[message]" ]
}

```

}

output {  
stdout { codec =\> rubydebug }  
elasticsearch {  
hosts =\> ["192.168.0.1:9200"]  
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2018, 9:27am UTC](https://discuss.elastic.co/t/split-nested-json-array/147969/10 "2018-10-09T09:27:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.