Discover query : timeout after a 99h treshold

Elastic Stack Elasticsearch
1

Hello,

I have one index a day in Elasticsearch (6.1) and using Kibana (with defaults parameters), I try to do a query on several days in the discover tab. It works for the 4 last days (99h exactly) and in the response, Elasticsearch says it took ~6s, hitting 12 000 000 events. The response is 1.3M characters, which means ~10Mbits.

With 5 days or more (or even 100 hours), I hit a 30s timeout. I don't know what is the issue here:

  • Kibana does not have any specific logs.
  • None of my elasticsearch servers seems to have any issue with any resources (heap, ram, cpu, io, network, load)
  • Whether I query for 2 days or 99 hours, there is no big changes in the time it takes (between 5 & 6 seconds)
  • 1.3Mo does not seem a lot to fit in RAM, the network speed is far more than 10Mbit/s.

Kibana launches the query on a local client node, which does not show any logs.

Is there some parameter I missed ? Where could I troubleshoot this issue ?

Thank you,
Regards,
Grégoire

2

Hello,

Is there someone who knows what I could miss ? Outside of CPU, RAM, IO, where could be the bottleneck ?

Regards,
Grégoire

3

What is the average size of your documents? Are you using the default value for discover:sampleSize under Advanced Settings? Have you tried setting doc_table:highlight to false under Advanced Settings to see if t his makes a difference?

4

The average size of documents is ~700 bytes. I kept the defaut value for discover:sampleSize, and indeed, disabling highlight enable me to get more data. Now, I still have performances issues but it does not seem to be a kibana one.

Is there some documentation to estimate the impact of highlight and plan how to size my kibana instances ?

Thank you,
Regards,
Grégoire