Split nested json array from log

Elastic Stack Logstash
1

I would like to to retrieve every element in below JSON to be a field so as to visualize in kibana by applying metrics in dashboard.

log
log1318×57 3.16 KB

I need each object in the array msg to be a separate entry in Elasticsearch and every attribute like eid etc to be a field. How would I create filter in configuring Logstash to do this?

I would like to have output to elasticsearch

{
"name":"server-app",
"hostname":"hostname11",
...........
...........
"msg.eid":"PRINT",
"msg.ver":"10.0",
........
.......
}

2

Use a json codec or a json filter.

3

I tried with below logstash.conf but no success :

input {
beats {
port => 5044
}
}

filter {
json { source => "message" }
json { source => "message" }
split { field => "msg" }

mutate {
	add_field => { 
		"EId" => "%{[msg][eid]}"
		"Ver" => "%{[msg][ver]}"
	}

}

}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
stdout{
codec => rubydebug
}
}

4

Please suggest how to resolve.

5

Should that second json filter be parsing [msg] rather than [message]?

6

I modified filter
filter {
json { source => "message" }
json { source => "msg" }
split { field => "msg" }

mutate {
	add_field => { 
		"EId" => "%{[msg][eid]}"
		"Ver" => "%{[msg][ver]}"
	}

}

}

but in Kibana still getting
EId %{[msg][eid]}
Ver %{[msg][ver]}

7

kibana is an awful tool to debug logstash problems with. It is just way too far downstream. You say you have

stdout { codec => rubydebug }

in your output section. What does the rubydebug output look like?

8

I logged the codecrubydebug output to file.

{
"message" => "{"name":"server-app","hostname":"hostname123","pid":66032,"level":30,"msg":"[{\"eid\":\"PRINT\",\"ver\":\"10.0\",\"context\":{\"channel\":\"web\",\"pdata\":{\"id\":\"emp-app\",\"ver\":\"1.0.0\",\"pid\":\"\"},\"sid\":\"session-id-1234567\",\"did\":\"mobile-device1234\",\"cdata\":,\"rollup\":{}},\"object\":{\"id\":\"content1\",\"ver\":\"content-version-1\"},\"tags\":,\"edata\":{\"type\":\"VIEW\",\"pageid\":\"Emp Detail\"}}]","time":"2020-04-19T08:28:52.894Z","v":0}",
"log" => {
"offset" => 449244,
"file" => {
"path" => "D:\logs\server.log"
}
},
"input" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_jsonparsefailure"
],
"ecs" => {
"version" => "1.4.0"
},
"time" => "2020-04-19T08:28:52.894Z",
"Ver" => "%{[msg][ver]}",
"@timestamp" => 2020-04-19T08:28:58.677Z,
"agent" => {
"id" => "f8f61c7c-1a8c-4cfd-af37-192d1e9b692c",
"type" => "filebeat",
"version" => "7.6.2",
"hostname" => "hostname123",
"ephemeral_id" => "282fcc12-865d-47c1-b557-f7f4bb446b67"
},
"msg" => "[{"eid":"PRINT","ver":"10.0","context":{"channel":"web","pdata":{"id":"emp-app","ver":"1.0.0","pid":""},"sid":"session-id-1234567","did":"mobile-device1234","cdata":,"rollup":{}},"object":{"id":"content1","ver":"content-version-1"},"tags":,"edata":{"type":"VIEW","pageid":"Emp Detail"}}]",
"pid" => 66032,
"host" => {
"architecture" => "x86_64",
"name" => "hostname123",
"hostname" => "hostname123",
"id" => "fg8b7a72-5e13-45b0-9410-a0862a94e727",
"os" => {
"version" => "10.0",
"platform" => "windows",
"name" => "Win 10",
"kernel" => "xxxxxx",
"family" => "windows"
}
},
"name" => "server-app",
"level" => 30,
"@version" => "1",
"v" => 0,
"hostname" => "hostname123",
"EId" => "%{[msg][eid]}"
}

To get ruby logs i modified my logstash.conf file as below
input {
beats {
port => 5044
}
}

filter {
json { source => "message" }
json { source => "msg" }
split { field => "msg" }

mutate {
	add_field => { 
		"EId" => "%{[msg][eid]}"
		"Ver" => "%{[msg][ver]}"
	}

}

}

output {

file{
path => "D:/logstash-7.6.2/logs/codecrubylog.log"
codec => rubydebug
}
}

9

Please suggest new logstash conf "filter" section if mine is not correct.

10

[msg] is an array so you must supply the target option on this filter.

11

Do you mean change like below?
input {
beats {
port => 5044
}
}

filter {
json { source => "message" }
json { source => "msg"
target => "parsedMsg"}
split { field => "parsedMsg" }

mutate {
	add_field => { 
		"EId" => "%{[parsedMsg][eid]}"
		"Ver" => "%{[parsedMsg][ver]}"
	}

}

}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
stdout{
codec => rubydebug
}
}

if yes, then its not working.

12

What does the rubydebug output look like with that configuration?

13

I am able to fix the issue. I am putting my file for other people in the community reference. I have modified as below
input {
beats {
port => 5044
codec => json {}
}
}

filter {

json { source => "msg" 
	   target => "parsedMsg"}
split { field => "parsedMsg" }

mutate {
	add_field => { 
		"EId" => "%{[parsedMsg][eid]}"
		"Ver" => "%{[parsedMsg][ver]}"
	}

}

}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
stdout{
codec => rubydebug
}
}

Thanks Badger for your support during analysis.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.