I have a strange problem.
We have an ES cluster with three nodes, each node's heap memory is 8 GB and the es version is 7.8.
Now I have two indexes: test@20210711125233 and test@20210712075544. There mappings are exactly the same, and their documents number is not much different(both around 26G with primary size). test@20210711125233 has 6 primary shards and test@20210712075544 has only 2 primary shards.
The number of segments for test@20210711125233 is 168 and test@20210712075544 has 53 segments.
Now I query these two indexes with same DSL, and test@20210712075544 took almost twice as long as test@20210711125233.
I was surprised that test@20210712075544 has fewer segments and fewer shards (I know that fewer shards is not always better, but considering the size of my index, two shards should be more suitable than 6 shards), why does its query take longer?
Here is some information in kibana:
I noticed that the latency of test@20210712075544 is much higher than test@20210711125233, but I am not sure if this is the cause of the problem.
In addition, what factors determine the index latency? How to effectively reduce the latency of indexing?