Data table don't display field data when a previous aggregation bucket field is empty

[jpertejo (Joaquin Pertejo)]

I have created a data table with several fields as aggregation buckets (e.g. "flow", "view" and "event" fields). Some of these fields are not informed in some documents (e.g. "view" event is not always informed), so I have activated "Show partial rows" option to display results with partial data. Then all rows with partial data are shown, but rows where "view" field is empty also don't display the value of "event" field, although "event" field is informed in all rows.

How can I configure data table for displaying all fields data even if some previous bucket field is empty?

[a5a (Archana)]

Joaquin,

I follow a bit what you're asking but would like more info. Can you share your configuration, i.e. a screenshot of the actual query in the sidebar? And perhaps also a sample request/response?

Depending on how you're structuring your query, you may be able to do enable min_doc_count: 0, which will return empty buckets. But seeing the actual aggregation you're using will help narrow it down a lot.

Thanks,
Archana (a5a)

[jpertejo (Joaquin Pertejo)]

Hi Archana,

Thanks for your response.

First of all I would like to explain you what is our goal, just in case we aren't resolving our requirement properly. What we are trying to do is showing a table with individual document data, based on a filter selected previously by user (filter by "transactionId" field) that should return a bunch of documents: usually between 1 and a few tens, in extraordinary cases they can reach a few hundred.

Our first solution was inserting a saved search into the dashboard and it shows all data correctly, but as I said in a previous post, saved searches don't allow costumizing column labels and our customer don't like it because they aren't consistent with the rest of the dashboard (and other dashboards).

As a workaround we have created a data table visualization that define aggregations and metrics for all fields that must be shown. The first aggregation bucket ("treeMap" field) is a unique identifier for documents with the same "transactionId" value, so it ensures that each row of the table is an individual document.

Below is the definition of the visualization. Note that some aggregation buckets are disabled and columns "Evento" and "Servicio" has data for all rows,

Then we enable the bucket aggregation by "targetComponentDesc". This field is not informed in document of first row (document with treeMap:1), but after enabling this aggregation columns "Evento" and "Servicio" aren't neither informed in first row:

If we also enable aggregation by "flow" field (this field isn't informed in any document for this "transactionId") "Evento" and "Servicio" aren't neither shown in any row:

Then we disable aggregation by "flow" and enable aggregation by "view". This field is informed in 6 documents with an empty string. Again "Evento" and "Servicio" columns are only shown for documents that has the "view" field.

The search query is simple:

{
"query": {
"match": {
"transactionId": {
"query": "0202eee1-71f5-47c6-901c-c1152ac1687d",
"type": "phrase"
}
}
}
}

I've tried to insert here the sample data used for this example, but then the post exceeds maximum allowed size. If you need to see row data I can send you by any other way.

I hope this information help you understand our issue.

Thanks,
Joaquin

[a5a (Archana)]

Hi Joaquin,

Thanks for sharing more about the problem. So, the reason you're not seeing the values in some cells, is because there is no data for those buckets at that point in the aggregation. To better understand this, you have to think of aggregations as a nested tree structure. To help you visualize what I'm talking, about, there's actually a "spy" view in Kibana that can help. See here at the bottom, the arrow icon that's highlighted:

Once you open that, select "Response" from the dropdown menu to view the raw Elasticsearch response.

Enable and disable the different aggregations now and see what's happening with your data.

I think what you might actually want is to use filters. The point in your aggregation chain at which you no longer have data for a document, that's where I would start using filters rather than nesting more aggregations.

So in your case, you have flow, view, event, service in a chain. Flow is where you start having incomplete info, so you could start your filters there.

The screenshot above shows a bit how you'd go about entering filters. Unfortunately we don't have the ability to change the Label for the column to be something other than "filters". Additionally, you'll have to hardcode values for this field, but looking at the values for your Event and Service fields, it looks like there is a finite set of values. So, for example, here are two filters:

And so forth.

Please take your time looking at these options and ask us any more questions.

-Archana

[jpertejo (Joaquin Pertejo)]

Hi Archana,

I have been playing around with your idea of ​​using filters, but unfortunately I don't think it will useful in our case. Flow, event and other fields can have a large number of different values that we don't know beforehand and they will also change over time, so hardcore values isn't a suitable solution.

Now we are considering two possible alternatives:

1. Use our first approach of inserting a saved search and try to convince our customer that it won't have custom labels for the moment.

2. Force that all documents will have all fields informed, even with an empty value.

Thanks,
Joaquin