Nested objects in Elasticsearch and Kibana

Hi,

I have a sample JSON file test.json with the following content:

{"name":"Jonathan","score":"9.9","address":"New Delhi","lastUpdated":null, "firstUpdated":"86400","official": [{"id": "ABCD1","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "1"}, {"id": "ABCD2","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "1"}]}
{"name":"Sam","score":"8.9","address":"New York","lastUpdated":"1545078074640", "firstUpdated":"86400","official": [{"id": "MNOP1","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "2"},{"id": "MNOP2","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "2"}]}
{"name":"Michelle","score":"9.0","address":"California","lastUpdated":"1545078074640", "firstUpdated":"86400","official": [{"id": "WXYZ1","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "3"},{"id": "WXYZ2","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "3"}]}

I want to parse this JSON data using logstash and visualize the same in Kibana. Since the JSON data has some nested fields, is there a way to parse them as it is i.e. In Kibana, I want the column official to be an array and have multiple items which in turn can have a nested structure.

My logstsash configuration is something like this: test.config

input{
file{
path => //path to the json file
codec => json
sincedb_path => "/dev/null"
start_position => "beginning"
}
}

filter{
json{
source => "student"
target => "student"
}

date{
match => ["lastUpdated", "UNIX_MS"]
target => "lastUpdated"
}

date{
match => ["firstUpdated", "UNIX_MS"]
target => "firstUpdated"
}

mutate{
convert => {
"name" => "string"
"score" => "float"
"address" => "string"
}
}

}

output{
elasticsearch{
hosts => "localhost:9200"
index => "test"
}
stdout { codec => rubydebug }
}

What should my logstash configuration be so that I can parse the nested objects. My current logstash configuration parses the JSON and this is what I see in Kibana's discover tab:

January 5th 2019, 23:56:20.093 name:Sam address:New York lastUpdated:December 18th 2018, 01:51:14.640 firstUpdated:January 1st 1970, 05:31:26.400 official:{ "rank": "Manager1", "uploader": { "AGS": 1544817662070, "AGM": 1544817662070 }, "id": "MNOP1" }, { "rank": "Manager2", "uploader": { "AGS": 1544817662070, "AGM": 1544817662070 }, "id": "MNOP2" } **@version:**1 path:/Users/amsing/Study/data/test.json score:8.9 @timestamp:January 5th 2019, 23:56:20.093 host:amsingh-macOS _id:wftEH2gBfDvVAWDzDsRK _type:doc _index:test _score: -

The problem with this is the json data in the official field is saved as a string and I cannot access the individual field in the json for analytics.

Do I have to create separate fields for each of the inner JSON elements to achieve this?

Also, what if the nesting in the original JSON file further deepens?

Please don't add unnecessary and unrelated tags to your topics.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.