Hi,
I have a sample JSON file test.json with the following content:
{"name":"Jonathan","score":"9.9","address":"New Delhi","lastUpdated":null, "firstUpdated":"86400","official": [{"id": "ABCD1","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "1"}, {"id": "ABCD2","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "1"}]}
{"name":"Sam","score":"8.9","address":"New York","lastUpdated":"1545078074640", "firstUpdated":"86400","official": [{"id": "MNOP1","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "2"},{"id": "MNOP2","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "2"}]}
{"name":"Michelle","score":"9.0","address":"California","lastUpdated":"1545078074640", "firstUpdated":"86400","official": [{"id": "WXYZ1","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "3"},{"id": "WXYZ2","uploader": {"AGS": 1544817662070,"AGM": 1544817662070},"rank": "3"}]}
I want to parse this JSON data using logstash and visualize the same in Kibana. Since the JSON data has some nested fields, is there a way to parse them as it is i.e. In Kibana, I want the column official to be an array and have multiple items which in turn can have a nested structure.
My logstsash configuration is something like this: test.config
input{
file{
path => //path to the json file
codec => json
sincedb_path => "/dev/null"
start_position => "beginning"
}
}filter{
json{
source => "student"
target => "student"
}date{
match => ["lastUpdated", "UNIX_MS"]
target => "lastUpdated"
}date{
match => ["firstUpdated", "UNIX_MS"]
target => "firstUpdated"
}mutate{
convert => {
"name" => "string"
"score" => "float"
"address" => "string"
}
}}
output{
elasticsearch{
hosts => "localhost:9200"
index => "test"
}
stdout { codec => rubydebug }
}
What should my logstash configuration be so that I can parse the nested objects. My current logstash configuration parses the JSON and this is what I see in Kibana's discover tab:
January 5th 2019, 23:56:20.093 name:Sam address:New York lastUpdated:December 18th 2018, 01:51:14.640 firstUpdated:January 1st 1970, 05:31:26.400 official:{ "rank": "Manager1", "uploader": { "AGS": 1544817662070, "AGM": 1544817662070 }, "id": "MNOP1" }, { "rank": "Manager2", "uploader": { "AGS": 1544817662070, "AGM": 1544817662070 }, "id": "MNOP2" } **@version:**1 path:/Users/amsing/Study/data/test.json score:8.9 @timestamp:January 5th 2019, 23:56:20.093 host:amsingh-macOS _id:wftEH2gBfDvVAWDzDsRK _type:doc _index:test _score: -
The problem with this is the json data in the official field is saved as a string and I cannot access the individual field in the json for analytics.
Do I have to create separate fields for each of the inner JSON elements to achieve this?
Also, what if the nesting in the original JSON file further deepens?