We're observing timeout while creating an index pattern in Kibana.
That means an aggregation query below takes over 30 seconds.
A curl command that runs the query above took around 40 seconds actually.
The ES cluster contains 650+ TB of data over 25,000+ primary shards, indexing 1+M documents/s constantly.
As suggested in a discussion thread, it can be acceptable to increase request timeout.
But, do we have any chance to improve the query performance?
Any chance to benefit from
Shard query time average: 528 ms, Shard query time max: 7,319 ms
Rewrite time average: 2.3 us, Rewrite time max: 1,550 us
Collect time average:1,642 ms, Collect time max: 9,589 ms
Aggregation time average: 671 ms, Aggregation time max: 16.5 s
- Increase data size per shard.