I have one index a day in Elasticsearch (6.1) and using Kibana (with defaults parameters), I try to do a query on several days in the discover tab. It works for the 4 last days (99h exactly) and in the response, Elasticsearch says it took ~6s, hitting 12 000 000 events. The response is 1.3M characters, which means ~10Mbits.
With 5 days or more (or even 100 hours), I hit a 30s timeout. I don't know what is the issue here:
- Kibana does not have any specific logs.
- None of my elasticsearch servers seems to have any issue with any resources (heap, ram, cpu, io, network, load)
- Whether I query for 2 days or 99 hours, there is no big changes in the time it takes (between 5 & 6 seconds)
- 1.3Mo does not seem a lot to fit in RAM, the network speed is far more than 10Mbit/s.
Kibana launches the query on a local client node, which does not show any logs.
Is there some parameter I missed ? Where could I troubleshoot this issue ?