Enhanced data table slow response in comparison to kibana discover search?

Elastic Stack Kibana
1

Hello @fbaligand ,

Can you please let me understand why enhanced table/document table provides data slow or sometime hangs when filtered for huge data.
Example 1 month data fetch,document count 20 million(here the data dosen't comes or page become unresponsive)
Meanwhile if same done through discover search it fetches data in fraction of seconds.

Through enhanced table simply I'm selecting various column values ex:15 column values showing in table and in index document count is around 20 million approx for 1 month time range.

We heavily depend on enhanced table due to its features and multiple usecases use this table.
I'm stuck to understand why discover works and not enhanced table for fast retrieval.discover gets data fast and enhanced table used then page become unresponsive.Even for count as small as 80000 it takes time.

First enhanced table is for aggreagtion-when used this for 1 million document count to show in table for few column value,page unresponsive.
Top hit(column1),Top hit(column2)....etc.....bucket by some_unique_id

Then used document table for other usecase where simply need to show 5 columns which are converted to hyperlink.this values are all unique and doc count was approx 10 million.page unresponsive.

image

ex issue :above image show few data ,consider it to be 10 million value that time page unresponsive,and similarly I have additional columns value .

Kibana version 7.9.1
ehanced-table version: 1.11.0

Am I doing something wrong or these tables are not meant for huge volume of data like discover search?

Again tnx this plugin helps a lot in terms of its features.

2

Hi,

First, as you say, Enhanced Table visualization is done for aggregations. So concretely, if your need is 1 row = 1 document, then Enhanced Table is not done for that.
If you do 15 columns in Enhanced Table, it means that you do 15 aggregations. And this is generally very cost-effective for Elastisearch itself. You can view Elasticsearch query time, if you click on "Inspect", and look at request statistics.

As "Discover" does a simple query without aggregation, you can't compare it to Enhanced Table, especially if you have 15 aggregations.

Then Document Table does a simple query with no aggregation. It is done to show one line = one document.
It has good performances if you display until 1000 documents and use pagination.
The big difference between Document Table and Discover is that Discover only gets and displays the first 500 rows. Not more. This is default value, that you can customize in Advanced Settings in Kibana.

If you require 10000 documents in Document Table, it will get from Elasticsearch the 10000 documents and store it in browser memory, and then display first N first documents (N is your table pagination size).
If you ask more than 10000 documents (in 'Hits size' option), you have to know that Document Table will download ALL documents in browser memory. That is clearly not recommended.
And Enhanced Table will do multiple queries to Elasticsearch. First query to get the first 10000 documents, then a second to get the following 10000 documents, and so on.
So if you really ask 10 million in 'Hits size' option, it can't work: you can load 10 million documents in browser memory.

I recommend you to use maximum 10000 documents in 'Hits size' when you use Document Table and use table pagination.

I hope that now you better understand how Enhanced Table and Document Table work and that my answer helps you.

3

Hello @fbaligand ,

Thanx for your timely response on this and time to answer in simplest way.
I can now relate diff between discover|enhanced table | document table.

I'd like to have another suggestion from you,if you see above image there is request id
for column,Now I've 1 million service request id to be searched for 5 days.
The reason to use enhanced table here is because I can convert them into downloadable links or route to another url on click.The same I'm not able to do with discover.
Though there is option of URL template in Index pattern in kibana,that dosent help,
because I'm implementing a solution where user won't be required to come to kibana for any changes i.e url template itself should be changed through some external config.

ex: https://www.randomdomain:3245/custompath/home,Here the host address and port would change frequently and user wont come to kibana to change this everytime if
host or port changes are required .

Now I see a challenge how huge volume of data could be searched using table.(Non aggregated data)

Again thanx for your time and efforts on this plugin.

Many Thanx

4

Well, if you want to search across a huge volume of data, I recommend you to use Kibana search capabilities. That is to say: Search Bar, Filter Bar or Control visualization (and not Document Table filter bar).

If you search for a specific field, Control visualization is probably the most convenient solution, especially if you use "Apply on each change" option.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.