The following issue is a problem with the structure of my data (Sample data has been used as a part of this example):
DOCUMENT STRUCTURE: We are using an array of objects (in this case it is vehicles) and the root of our document in this case is driver. (This below document structure shows implementation as a nested object but we have tried array of objects, nested type and parent-child. The issues with all will be discussed in the end)
PUT /drivers
{
"mappings" : {
"properties" : {
"driver" : {
"type" : "nested",
"properties" : {
"last_name" : {
"type" : "text"
},
"vehicle" : {
"type" : "nested",
"properties" : {
"make" : {
"type" : "text"
},
"model" : {
"type" : "text"
}
}
}
}
}
}
}
}
SAMPLE DATA: Following would be a sample data into the Elastic search
"driver" : {
"last_name" : "McQueen",
"vehicle" : [
{
"make" : "Powell Motors",
"model" : "Canyonero"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
"driver" : {
"last_name" : "Hudson",
"vehicle" : [
{
"make" : "Mifune",
"model" : "Mach Five"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
SAMPLE QUERY: In Kibana visuals, I would like to aggregate over the count of drivers that will have a car with a particular make/model (make or model will be a filter in Kibana). However, we are having certain issues with the filtering of data with the different methods that we have tried:
-
Nested Objects- By using the nested objects, I am not able to perform any aggregations in Kibana and it makes sense from the documentation by elastic here. So I cannot perform the visuals if I set the vehicles as a nested object under the drivers.
-
Driver-Vehicles as Parent-Child OR Vehicles as Array of objects under Driver: By using both of these methods, I am able to perform aggregation on the objects. However, if we filter on the vehicle model as Canyonero the returned document (as per the 2 samples given above ) is the root document at driver level, not the vehicle level. Following will be the returned object:
"driver" : {
"last_name" : "McQueen",
"vehicle" : [
{
"make" : "Powell Motors",
"model" : "Canyonero"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
The issue with returning the root level document here is that we still see the model Ecto-1 in our visuals even if we have filtered on the Canyonero, which is not good for our dashboards and visuals. What we want is that the returned document should look like following:
"driver" : {
"last_name" : "McQueen",
"vehicle" : [
{
"make" : "Powell Motors",
"model" : "Canyonero"
}
]
}
This way we get back what we wanted as out filtered object.
- Flatten Objects: I have also tried to flatten out the document such that there will be a root document for each vehicle. That way there are multiple documents for a driver with multiple cars, which is creating issues for the counts that we are doing in our logic of documents.
I was hoping someone could suggest a suitable method to deal with this issue? Please feel free to ask any follow up question that you may have understanding the requirements.