Data structure / type help

The following issue is a problem with the structure of my data (Sample data has been used as a part of this example):

DOCUMENT STRUCTURE: We are using an array of objects (in this case it is vehicles) and the root of our document in this case is driver. (This below document structure shows implementation as a nested object but we have tried array of objects, nested type and parent-child. The issues with all will be discussed in the end)

PUT /drivers
{
    "mappings" : {
        "properties" : {
            "driver" : {
                "type" : "nested",
                "properties" : {
                    "last_name" : {
                        "type" : "text"
                    },
                    "vehicle" : {
                        "type" : "nested",
                        "properties" : {
                            "make" : {
                                "type" : "text"
                            },
                            "model" : {
                                "type" : "text"
                            }
                        }
                    }
                }
            }
        }
    }
}

SAMPLE DATA: Following would be a sample data into the Elastic search

"driver" : {
        "last_name" : "McQueen",
        "vehicle" : [
            {
                "make" : "Powell Motors",
                "model" : "Canyonero"
            },
            {
                "make" : "Miller-Meteor",
                "model" : "Ecto-1"
            }
        ]
    }

"driver" : {
        "last_name" : "Hudson",
        "vehicle" : [
            {
                "make" : "Mifune",
                "model" : "Mach Five"
            },
            {
                "make" : "Miller-Meteor",
                "model" : "Ecto-1"
            }
        ]
    }

SAMPLE QUERY: In Kibana visuals, I would like to aggregate over the count of drivers that will have a car with a particular make/model (make or model will be a filter in Kibana). However, we are having certain issues with the filtering of data with the different methods that we have tried:

  1. Nested Objects- By using the nested objects, I am not able to perform any aggregations in Kibana and it makes sense from the documentation by elastic here. So I cannot perform the visuals if I set the vehicles as a nested object under the drivers.

  2. Driver-Vehicles as Parent-Child OR Vehicles as Array of objects under Driver: By using both of these methods, I am able to perform aggregation on the objects. However, if we filter on the vehicle model as Canyonero the returned document (as per the 2 samples given above ) is the root document at driver level, not the vehicle level. Following will be the returned object:

"driver" : {
        "last_name" : "McQueen",
        "vehicle" : [
            {
                "make" : "Powell Motors",
                "model" : "Canyonero"
            },
            {
                "make" : "Miller-Meteor",
                "model" : "Ecto-1"
            }
        ]
    }

The issue with returning the root level document here is that we still see the model Ecto-1 in our visuals even if we have filtered on the Canyonero, which is not good for our dashboards and visuals. What we want is that the returned document should look like following:

"driver" : {
        "last_name" : "McQueen",
        "vehicle" : [
            {
                "make" : "Powell Motors",
                "model" : "Canyonero"
            }
        ]
    }

This way we get back what we wanted as out filtered object.

  1. Flatten Objects: I have also tried to flatten out the document such that there will be a root document for each vehicle. That way there are multiple documents for a driver with multiple cars, which is creating issues for the counts that we are doing in our logic of documents.

I was hoping someone could suggest a suitable method to deal with this issue? Please feel free to ask any follow up question that you may have understanding the requirements.

The only supported way to do this is to change your index to contain data in the shape you want to visualize, nested/parent-child relationships are not supported in general in kibana: https://www.elastic.co/guide/en/kibana/current/nested-objects.html

However, you may be able to use the a Vega visualization using the full Query DSL to work around these restrictions. There are limitations to this, which is why I would recommend going with your 3rd option.

Hi @wylie,

Thanks for your reply. I am looking at Vega as a solution now. I am a bit concern as this functionality is experimental, but I think it could fix my solution... just need to learn the syntax/tool a bit more.

I think the 3rd option is the best, but struggling on setting up the document structure.