Kibana freezing when creating TSVB

Elastic Stack Kibana
1

Hello,

we have an elk cluster from 5 ES nodes, xpack security enabled basic license.
whenever we try to create a TSVB (previously Visual Builder) visualization Kibana freezes and the java node process shows high CPU consumption over 100%. We tried to wait for more then 15 min but the freezing and the high cpu persist. No significant error in kibana logs were observed , knowing that the verbosity logging is enabled. The problem was first seen on version on version 7.2 ( visual Builder) , And upgrading the entire cluster (ES, kibana) to 7.3 did not solve the problem.
We stopped the logstash for 15 min to rule out the possibility of having huge data that was causing the TSVB to freeze, but same behavior was observed .

2

Hey @pinguin,

Just a couple of followup questions for you:

  • Does this happen with every TSVB visualization you attempt?
  • How large are the indices you're working with?
  • What type of TSVB visualization are you attempting?
  • If you create an index with only a handful of documents, are you able to create a TSVB visualization off of that one?
3

hello larry ,

thank u for your help,
1-First this happen whenever we try to click on Create new Visualization , and then we choose TSVB it will do the explained behavior.
2-Before choosing the index the error occurs
3-Before choosing the type the error occurs
4- We tried to stop all the logs for 15 min since the default time of TSVB time range is 15 min, so we have like 0 docs on all our index during this 15 min , but still it hangs for 15 min.
5- we tried all other visualization one by one all working like charm except the TSVB.

Kindly find the attached image , when clicking on the below it crash for 15 min to 20 min

image
image.png1376×726 104 KB

4

Thanks for the update @pinguin.

When loading the TSVB editor, a request is made to Elasticsearch which grabs the available fields from all indices. I suspect that this request is what's causing your cluster to have a CPU spike and generally become unresponsive.

Do you have a lot of indices, or indices with many many fields defined in their index mappings?

What happens if you issue the following request to Elasticsearch directly? How long does this request take to complete? Do you see the same performance drop?

GET /_field_caps?fields=*&ignore_unavailable=true&allow_no_indices=false
5

hello Larry,

The command took about 4 min during those 4 min the system hanged.
we have 593 indices and over all 3808 fields in all indices

(my cluster is 120 GB RAM , AND kibana is 24 CORE CPU)

6

Thanks for running this test. It does seem like this particular call is at least part of the cause then. TSVB uses this call behind the scenes when loading the visual editor.

Do you use cross-cluster search by chance?

7

Dear Larry,

we don't use cross-cluster search.

8

Dear larry

do u want me to provide u with some logs , or enable more logging level to get more info please?

9

Hey @pinguin,

I've opened a GitHub issue to track and address this: https://github.com/elastic/kibana/issues/43769

10

thank you so much for your help !!!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.