Nested Objects in a json Response displayed in Kibnana


(Giridharan Manoharan) #1

Continuing the discussion from I would like to access nested data from an array:

The data inside the array are not getting indexed and hence not displayed on the kibana dashboard. What is the best way to process the data from Logstash > elastic search > kibana. I wanted to get the data in the results and get them displayed in kibana. Sample of my data below. Presently the data is displayed as single object on kibana (discover)
Any suggestions would be very much appreciated.

{
"meta":{
"paginationTotalResults": 3,
"response_in_sec": 0.08
},
"results":[
{
"pageTitle": "Breast cancer | Cancer Research UK",
"url": "http://www.cancerresearchuk.org/about-cancer/type/breast-cancer/",
"browserUri": "browsers/ff",
"dataStartDuration": 0.067,
"renderStart": 3.071
}
{
{
"pageTitle": "Breast cancer | Cancer Research UK",
"url": "http://www.cancerresearchuk.org/about-cancer/type/breast-cancer/",
"browserUri": "browsers/ff",
"dataStartDuration": 0.094,
"renderStart": 2.266
}
]
}

Config file used in logstash
input {
http_poller{
urls => {
test1 => {
method => get
url => "https://xxxxxxxx.com/pa/1/testRuns?jobTemplate=jobTemplates/76045"
headers => {
Authorization => "Bearer c59e764ccad6ffc0a5ca28eb50255ae910b126190baa5e020e9da5403660"
Realm => "626367"
}
}
}

request_timeout => 1200
interval => 1200
codec => json
# A hash of request metadata info (timing, response headers, etc.) will be sent here
#metadata_target => "http_poller_metadata"
}
}
filter{

}
output {
#stdout { codec => ruby_debug }
elasticsearch {
hosts => "http://localhost:9200"
index => "api-index"
}
}


(Magnus Bäck) #2

Kibana doesn't fully support arrays of objects.

The data inside the array are not getting indexed and hence not displayed on the kibana dashboard.

What, exactly, do you mean by this? ES has no problem indexing documents with arrays of objects.


(Giridharan Manoharan) #3

Sorry, my understanding might be wrong. I thought if all the objects inside the array are indexed then there would be no problem getting them displayed on kibana. The main problem is getting the data displayed on Kibana. Followed all the process mentioned in https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html.

Could you please advise what would be the best way to display the nested objects on kibana.


(Magnus Bäck) #4

I suggest you ask in the Kibana category, but as I said Kibana's handling of arrays of objects is a known limitation.


(Giridharan Manoharan) #5

Found a solution. Split filter has worked.

Reference: https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.