Split nested json array

[slavo3 (Slawek)]

I would like to to retrieve every element in below JSON to be a field so as to visualize in kibana by applying metrics in dashboard.

{
"message":
"{"SegmentId":4,"VectorList":[{"LocalizationId":60,"ServiceId":1,"Date":"2018-09-10T00:00:00","IsAvailable":true,"FreeTermCount":80,"SegmentFreeTermCount":80,"FilterValue":null},{"LocalizationId":60,"ServiceId":2,"Date":"2018-09-10T00:00:00","IsAvailable":true,"FreeTermCount":21,"SegmentFreeTermCount":21,"FilterValue":42},{"LocalizationId":60,"ServiceId":1,"Date":"2018-09-11T00:00:00","IsAvailable":true,"FreeTermCount":114,"SegmentFreeTermCount":114,"FilterValue":null},{"LocalizationId":60,"ServiceId":2,"Date":"2018-09-11T00:00:00","IsAvailable":true,"FreeTermCount":104,"SegmentFreeTermCount":78,"FilterValue":null}]}"
}

I need each object in the array VectorList to be a separate entry in Elasticsearch and every attribute like LocalizationId etc to be a field. How would I create filter in configuring Logstash to do this?

I know that i sould use split filter, but my attempts failed. Is there anyone who can help me define the filter?

I can add that message value was created by wcf service using Json.NET - Newtonsoft so characters like " are generated automatically.

[magnusbaeck (Magnus Bäck)]

The general idea is to use a json filter to parse the JSON string in the message field and then use a split filter to split the array in the VectorList field we got from the json filter.

If you show us what you have so far it'll be easier to help.

[slavo3 (Slawek)]

I would like to have output to elasticsearch

{
"SegmentId":4,
"LocalizationId":60,
"ServiceId":1,
"Date":"2018-09-10T00:00:00",
"IsAvailable":true,
"FreeTermCount":80,
"SegmentFreeTermCount":80,
"FilterValue":null
}

{
"LocalizationId":60,
"ServiceId":2,
"Date":"2018-09-10T00:00:00",
"IsAvailable":true,
"FreeTermCount":21,
"SegmentFreeTermCount":21,
"FilterValue":42
}
....

[magnusbaeck (Magnus Bäck)]

Yes, I understand that. I'm interested in what your Logstash configuration looks like so far.

[slavo3 (Slawek)]

filter {

    split { 
        field => "message"
    }
	
	 mutate {
		add_field => { 
			"LocalizationId" => "%{[message][VectorList][ServiceVariantId]"
			"ServiceVariantId" => "%{[message][VectorList][ServiceVariantId]}"
			"Date" => "%{[message][VectorList][Date]}"
			"IsAvailable" => "%{[message][VectorList][IsAvailable]}"
			"FreeTermCount" => "%{[message][VectorList][FreeTermCount]}"
			"SegmentFreeTermCount" => "%{[message][VectorList][SegmentFreeTermCount]}"
			"FilterValue" => "%{[message][VectorList][FilterValue]}"
}

remove_field => [ "[message]" ]

}

[magnusbaeck (Magnus Bäck)]

Insert a json { source => "message" } filter prior to your split filter and change the split filter's field option to "VectorList".

[slavo3 (Slawek)]

Thanks for advice,

my single row in file:
{"message":"{SegmentId:3,VectorList:[{LocalizationId:60,ServiceId:2,Date:2018-09-11T00:00:00,IsAvailable:true,FreeTermCount:92,SegmentFreeTermCount:92,FilterValue:null},{LocalizationId:60,ServiceVariantId:2,Date:2018-09-12T00:00:00,IsAvailable:true,FreeTermCount:48,SegmentFreeTermCount:34,FilterValue:68},]}"}

I would like you to pay attention to the format
{"message":"{ ...}"

and when i added
filter {
json { source => "message" }
...
}
i received error parsing json:

How can i resolve it?

[slavo3 (Slawek)]

OK, i repaired my single row and now parsing is ok

{"message":"{"SegmentId":3,"VectorList":[{"LocalizationId":60,"ServiceId":2,"Date":"2018-09-11T00:00:00","IsAvailable":true,"FreeTermCount":86,"SegmentFreeTermCount":86,"FilterValue":null},{"LocalizationId":60,"ServiceId":2,"Date":"2018-09-12T00:00:00","IsAvailable":true,"FreeTermCount":48,"SegmentFreeTermCount":34,"FilterValue":68}]}"}

[slavo3 (Slawek)]

[Magnus Bäck] Thanks for your help. Everythink is ok, below is my logstash configuration which is correct and generate event per element in my vectorList.

input {
file {
type => "MyLog"
path => ["C:/logs/X/*.json"]
start_position => "beginning"
codec => json
}
}

filter {
json { source => "message" }

split { 
    field => "VectorList"
}

mutate {
	add_field => { 
		"LocalizationId" => "%{[VectorList][ServiceVariantId]}"
		"ServiceVariantId" => "%{[VectorList][ServiceVariantId]}"
		"Date" => "%{[VectorList][Date]}"
		"IsAvailable" => "%{[VectorList][IsAvailable]}"
		"FreeTermCount" => "%{[VectorList][FreeTermCount]}"
		"SegmentFreeTermCount" => "%{[VectorList][SegmentFreeTermCount]}"
		"FilterValue" => "%{[VectorList][FilterValue]}"
	}
	remove_field => [ "[message]" ]
}

}

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.0.1:9200"]
}
}