Hi,
I noticed a huge performance decrease when it comes to aggregations on multiple indices compared to aggregations one index at a time.
I got the following two indices:
- indexA ~ 1,250,000,000 docs (8 shards)
- indexB ~ 89,000,000 docs (4 shards)
My aggregation (which one doesnt really matter here) runs ~ 2100ms on indexA and ~1400ms on indexB.
When I run the same aggregation on an Alias which points to both indices, it runs ~20000ms.
I notices the same type of decrease for any type of aggregation I run (simple terms, cardinality etc).
Is this expected? Maybe due to the coordination work that needs to be done to merge those results together?
I mostly just split my indiced because it makes changes over time easier and because I didnt expect such a hit in terms of performance when querying multiple indices.
Should I try to put all my data into one big Index in order to avoid this kind of issues?
Another stange thing is, that I can not see any issues with regards to hot threads.
There is no thread which e.g. runs the same thing for the complete 'interval'.
Cheers,
Robert