Kibana freezing when creating TSVB

pinguin · August 19, 2019, 1:00pm

Hello,

we have an elk cluster from 5 ES nodes, xpack security enabled basic license.
whenever we try to create a TSVB (previously Visual Builder) visualization Kibana freezes and the java node process shows high CPU consumption over 100%. We tried to wait for more then 15 min but the freezing and the high cpu persist. No significant error in kibana logs were observed , knowing that the verbosity logging is enabled. The problem was first seen on version on version 7.2 ( visual Builder) , And upgrading the entire cluster (ES, kibana) to 7.3 did not solve the problem.
We stopped the logstash for 15 min to rule out the possibility of having huge data that was causing the TSVB to freeze, but same behavior was observed .

Larry_Gregory · August 19, 2019, 2:39pm

Hey @pinguin,

Just a couple of followup questions for you:

Does this happen with every TSVB visualization you attempt?
How large are the indices you're working with?
What type of TSVB visualization are you attempting?
If you create an index with only a handful of documents, are you able to create a TSVB visualization off of that one?

pinguin · August 19, 2019, 2:54pm

hello larry ,

thank u for your help,
1-First this happen whenever we try to click on Create new Visualization , and then we choose TSVB it will do the explained behavior.
2-Before choosing the index the error occurs
3-Before choosing the type the error occurs
4- We tried to stop all the logs for 15 min since the default time of TSVB time range is 15 min, so we have like 0 docs on all our index during this 15 min , but still it hangs for 15 min.
5- we tried all other visualization one by one all working like charm except the TSVB.

Kindly find the attached image , when clicking on the below it crash for 15 min to 20 min

Larry_Gregory · August 20, 2019, 5:58pm

Thanks for the update @pinguin.

When loading the TSVB editor, a request is made to Elasticsearch which grabs the available fields from all indices. I suspect that this request is what's causing your cluster to have a CPU spike and generally become unresponsive.

Do you have a lot of indices, or indices with many many fields defined in their index mappings?

What happens if you issue the following request to Elasticsearch directly? How long does this request take to complete? Do you see the same performance drop?

GET /_field_caps?fields=*&ignore_unavailable=true&allow_no_indices=false

pinguin · August 20, 2019, 6:46pm

hello Larry,

The command took about 4 min during those 4 min the system hanged.
we have 593 indices and over all 3808 fields in all indices

(my cluster is 120 GB RAM , AND kibana is 24 CORE CPU)

Larry_Gregory · August 20, 2019, 7:08pm

Thanks for running this test. It does seem like this particular call is at least part of the cause then. TSVB uses this call behind the scenes when loading the visual editor.

Do you use cross-cluster search by chance?

pinguin · August 20, 2019, 7:25pm

Dear Larry,

we don't use cross-cluster search.

pinguin · August 21, 2019, 10:16pm

Dear larry

do u want me to provide u with some logs , or enable more logging level to get more info please?

Larry_Gregory · August 22, 2019, 1:09pm

Hey @pinguin,

I've opened a GitHub issue to track and address this: https://github.com/elastic/kibana/issues/43769

pinguin · August 24, 2019, 8:59am

thank you so much for your help !!!

system · September 21, 2019, 8:59am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.