How to analyse nested fields?

[Amine16 (Chaabouni)]

I am working in e-health project and we have a lot of clinical data (JSON format) stored in our data base.
We want to do some research in these data, that’s why we think that Elastic Search tool can help us in this project.

I did some tests using kibana and ES (current version). Unfortunately, the results are not helpful. I need your help to be sure that I am using ES correctly.

In fact, I created 2 new indices (for 2 patients) containing some nested fields. For example, a patient has different consultations (nested field). Each consultation can have different MRI (nested field). And finally each MRI has a volume and a date field (simple fields).
Then I created a data view based on this index pattern.

In Discover Analytics, I want to display for example all MRI dates for all MRI volumes equal to 15.
What I did:

• In KQL syntax, I put consultation.MRI:{ volumeFlair : 15 } and I selected consultation.MRI.date. In Field statistics, I have 1hit result but I can't display the selected field (consultation.MRI.date). It is noted that Analysis is not available for this field.

If my example is not clear, we can take the example used in ES website (https://www.elastic.co/guide/en/elasticsearch/reference/8.6/nested.html)
In this example, I want to display the user.first field when user.last equal to Smith. I should have John as a result.
However, as the user is a nested field, I can't do this analysis!

As I understood, analysis is not available for nested field. Is it right?
In this case, I can’t analyse nested fields which it is a problem for our projet.

Do you have an idea about this problem and how to fix it?

[Amine16 (Chaabouni)]

I had an answer from Customer Success Engineering team of ES. It confirms that nested field cannot be use as 'analysis' in Kibana Discover. This means that we cannot directly build visualizations based on Nested fields directly from the Discover dashboard.
Two solutions recommended:

1/ The nested fields can be queried or returned as search results, and we can also build aggregations on top of it. The only drawback here is that nested queries might be slower, and visualizations involving nested fields need to be built in Kibana using vega custom visualization. It is not as straightforward as building using Lens, but there is a step by step tutorial in the shared link on how to do so. In short it can still be analysed in any way we would need the nested fields for.
==> This solution seems to be complicated to build.

2/ An alternative is to restructure the JSON into a flatter one, which we can then build a search experience without using nested operations. This will create more efficiency in the long term as the search operations will be significantly cheaper.
==> this solution may be easier. I used a tool to flat the nested JSON and I had a lot of fields for each field like: "consultation_1_IRM_0_volumeFlair" and "consultation_2_IRM_0_volumeFlair" ...
In Kibana, I should select all fields in relation with IRM.volumeFlair to do the analyse which it is a bad method. Each field will be treated separately while all consultation_X_IRM_X_volumeFlair fields refer to the same field which is consultation.IRM.volumeFlair.

The last solution is to generate different JSON files from one JSON file. In my example, I will generate MRI JSON file, treatement JSON file ex… for each patient JSON file. Which is not an optimized solution!

Let's take again the example of group/user used in ES website (Nested field type | Elasticsearch Guide [8.6] | Elastic).
In this case, we will generate a JSON file for GROUPs and a JSON file for USERs. In consequence, we will have two indexes: my_index_000001_group and my_index_000001_user.

How to connect these two indexes in ES? Maybe by using an aliase or we can generate a common field like an ID?

How to generate a Kibana visualisation including two fields (one field from my_index_000001_group and one field from my_index_000001_user)?

For this example, is it possible to display all the user.first for group : “fans” using Kibana Discover tool?

Thank you in advance for your help!