# Split nested json array from log

**URL:** <https://discuss.elastic.co/t/split-nested-json-array-from-log/228668>\
**Category:** Logstash\
**Created:** [April 18, 2020, 5:53pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668 "2020-04-18T17:53:57Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![rajender](https://avatars.discourse-cdn.com/v4/letter/r/e19adc/32.png) [@rajender](https://discuss.elastic.co/u/rajender)\
**Post date:** [April 18, 2020, 5:53pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/1 "2020-04-18T17:53:57Z")

</div>

I would like to to retrieve every element in below JSON to be a field so as to visualize in kibana by applying metrics in dashboard.

 ![log](https://us1.discourse-cdn.com/elastic/original/3X/0/4/04fb7ed7998cfecb5147a16339557f1044cf11ba.png)

I need each object in the array msg to be a separate entry in Elasticsearch and every attribute like eid etc to be a field. How would I create filter in configuring Logstash to do this?

I would like to have output to elasticsearch

{  
"name":"server-app",  
"hostname":"hostname11",  
...........  
...........  
"msg.eid":"PRINT",  
"msg.ver":"10.0",  
........  
.......  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 18, 2020, 6:34pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/2 "2020-04-18T18:34:54Z")

</div>

Use a json codec or a json filter.

---

<div class="post-metadata">

**Author:** ![rajender](https://avatars.discourse-cdn.com/v4/letter/r/e19adc/32.png) [@rajender](https://discuss.elastic.co/u/rajender)\
**Post date:** [April 18, 2020, 6:53pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/3 "2020-04-18T18:53:52Z")

</div>

I tried with below logstash.conf but no success :

input {  
beats {  
port =\> 5044  
}  
}

filter {  
json { source =\> "message" }  
json { source =\> "message" }  
split { field =\> "msg" }

```
mutate {
	add_field => { 
		"EId" => "%{[msg][eid]}"
		"Ver" => "%{[msg][ver]}"
	}

}

```

}

output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
}  
stdout{  
codec =\> rubydebug  
}  
}

---

<div class="post-metadata">

**Author:** ![rajender](https://avatars.discourse-cdn.com/v4/letter/r/e19adc/32.png) [@rajender](https://discuss.elastic.co/u/rajender)\
**Post date:** [April 18, 2020, 6:56pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/4 "2020-04-18T18:56:07Z")

</div>

Please suggest how to resolve.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 18, 2020, 8:49pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/5 "2020-04-18T20:49:51Z")

</div>

Should that second json filter be parsing [msg] rather than [message]?

---

<div class="post-metadata">

**Author:** ![rajender](https://avatars.discourse-cdn.com/v4/letter/r/e19adc/32.png) [@rajender](https://discuss.elastic.co/u/rajender)\
**Post date:** [April 18, 2020, 10:53pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/6 "2020-04-18T22:53:27Z")

</div>

I modified filter  
filter {  
json { source =\> "message" }  
json { source =\> "msg" }  
split { field =\> "msg" }

```
mutate {
	add_field => { 
		"EId" => "%{[msg][eid]}"
		"Ver" => "%{[msg][ver]}"
	}

}

```

}

but in Kibana still getting  
EId %{[msg][eid]}  
Ver %{[msg][ver]}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2020, 12:54am UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/7 "2020-04-19T00:54:11Z")

</div>

> [@rajender](#):
>
> in Kibana still getting

kibana is an awful tool to debug logstash problems with. It is just way too far downstream. You say you have

```
stdout { codec => rubydebug }

```

in your output section. What does the rubydebug output look like?

---

<div class="post-metadata">

**Author:** ![rajender](https://avatars.discourse-cdn.com/v4/letter/r/e19adc/32.png) [@rajender](https://discuss.elastic.co/u/rajender)\
**Post date:** [April 19, 2020, 9:07am UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/8 "2020-04-19T09:07:38Z")

</div>

I logged the codecrubydebug output to file.

{  
"message" =\> "{"name":"server-app","hostname":"hostname123","pid":66032,"level":30,"msg":"[{\"eid\":\"PRINT\",\"ver\":\"10.0\",\"context\":{\"channel\":\"web\",\"pdata\":{\"id\":\"emp-app\",\"ver\":\"1.0.0\",\"pid\":\"\"},\"sid\":\"session-id-1234567\",\"did\":\"mobile-device1234\",\"cdata\":,\"rollup\":{}},\"object\":{\"id\":\"content1\",\"ver\":\"content-version-1\"},\"tags\":,\"edata\":{\"type\":\"VIEW\",\"pageid\":\"Emp Detail\"}}]","time":"2020-04-19T08:28:52.894Z","v":0}",  
"log" =\> {  
"offset" =\> 449244,  
"file" =\> {  
"path" =\> "D:\logs\server.log"  
}  
},  
"input" =\> {  
"type" =\> "log"  
},  
"tags" =\> [  
[0] "beats\_input\_codec\_plain\_applied",  
[1] "\_jsonparsefailure"  
],  
"ecs" =\> {  
"version" =\> "1.4.0"  
},  
"time" =\> "2020-04-19T08:28:52.894Z",  
"Ver" =\> "%{[msg][ver]}",  
"@timestamp" =\> 2020-04-19T08:28:58.677Z,  
"agent" =\> {  
"id" =\> "f8f61c7c-1a8c-4cfd-af37-192d1e9b692c",  
"type" =\> "filebeat",  
"version" =\> "7.6.2",  
"hostname" =\> "hostname123",  
"ephemeral\_id" =\> "282fcc12-865d-47c1-b557-f7f4bb446b67"  
},  
"msg" =\> "[{"eid":"PRINT","ver":"10.0","context":{"channel":"web","pdata":{"id":"emp-app","ver":"1.0.0","pid":""},"sid":"session-id-1234567","did":"mobile-device1234","cdata":,"rollup":{}},"object":{"id":"content1","ver":"content-version-1"},"tags":,"edata":{"type":"VIEW","pageid":"Emp Detail"}}]",  
"pid" =\> 66032,  
"host" =\> {  
"architecture" =\> "x86\_64",  
"name" =\> "hostname123",  
"hostname" =\> "hostname123",  
"id" =\> "fg8b7a72-5e13-45b0-9410-a0862a94e727",  
"os" =\> {  
"version" =\> "10.0",  
"platform" =\> "windows",  
"name" =\> "Win 10",  
"kernel" =\> "xxxxxx",  
"family" =\> "windows"  
}  
},  
"name" =\> "server-app",  
"level" =\> 30,  
"@version" =\> "1",  
"v" =\> 0,  
"hostname" =\> "hostname123",  
"EId" =\> "%{[msg][eid]}"  
}

To get ruby logs i modified my logstash.conf file as below  
input {  
beats {  
port =\> 5044  
}  
}

filter {  
json { source =\> "message" }  
json { source =\> "msg" }  
split { field =\> "msg" }

```
mutate {
	add_field => { 
		"EId" => "%{[msg][eid]}"
		"Ver" => "%{[msg][ver]}"
	}

}

```

}

output {

file{  
path =\> "D:/logstash-7.6.2/logs/codecrubylog.log"  
codec =\> rubydebug  
}  
}

---

<div class="post-metadata">

**Author:** ![rajender](https://avatars.discourse-cdn.com/v4/letter/r/e19adc/32.png) [@rajender](https://discuss.elastic.co/u/rajender)\
**Post date:** [April 19, 2020, 9:09am UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/9 "2020-04-19T09:09:07Z")

</div>

Please suggest new logstash conf "filter" section if mine is not correct.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2020, 3:02pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/10 "2020-04-19T15:02:30Z")

</div>

> [@rajender](#):
>
> json { source =\> "msg" }

[msg] is an array so you must supply the target option on this filter.

---

<div class="post-metadata">

**Author:** ![rajender](https://avatars.discourse-cdn.com/v4/letter/r/e19adc/32.png) [@rajender](https://discuss.elastic.co/u/rajender)\
**Post date:** [April 19, 2020, 5:36pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/11 "2020-04-19T17:36:34Z")

</div>

> [@Badger](#):
>
> ion on this

Do you mean change like below?  
input {  
beats {  
port =\> 5044  
}  
}

filter {  
json { source =\> "message" }  
json { source =\> "msg"  
target =\> "parsedMsg"}  
split { field =\> "parsedMsg" }

```
mutate {
	add_field => { 
		"EId" => "%{[parsedMsg][eid]}"
		"Ver" => "%{[parsedMsg][ver]}"
	}

}

```

}

output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
}  
stdout{  
codec =\> rubydebug  
}  
}

if yes, then its not working.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2020, 7:49pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/12 "2020-04-19T19:49:07Z")

</div>

What does the rubydebug output look like with that configuration?

---

<div class="post-metadata">

**Author:** ![rajender](https://avatars.discourse-cdn.com/v4/letter/r/e19adc/32.png) [@rajender](https://discuss.elastic.co/u/rajender)\
**Post date:** [April 20, 2020, 12:42pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/13 "2020-04-20T12:42:31Z")

</div>

I am able to fix the issue. I am putting my file for other people in the community reference. I have modified as below  
input {  
beats {  
port =\> 5044  
codec =\> json {}  
}  
}

filter {

```
json { source => "msg" 
	   target => "parsedMsg"}
split { field => "parsedMsg" }

mutate {
	add_field => { 
		"EId" => "%{[parsedMsg][eid]}"
		"Ver" => "%{[parsedMsg][ver]}"
	}

}

```

}

output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
}  
stdout{  
codec =\> rubydebug  
}  
}

Thanks Badger for your support during analysis.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 18, 2020, 12:42pm UTC](https://discuss.elastic.co/t/split-nested-json-array-from-log/228668/14 "2020-05-18T12:42:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.