Aggregations on multiple indices much slower than on single index


(Robert Krombholz) #1

Hi,

I noticed a huge performance decrease when it comes to aggregations on multiple indices compared to aggregations one index at a time.
I got the following two indices:

  • indexA ~ 1,250,000,000 docs (8 shards)
  • indexB ~ 89,000,000 docs (4 shards)

My aggregation (which one doesnt really matter here) runs ~ 2100ms on indexA and ~1400ms on indexB.
When I run the same aggregation on an Alias which points to both indices, it runs ~20000ms.
I notices the same type of decrease for any type of aggregation I run (simple terms, cardinality etc).

Is this expected? Maybe due to the coordination work that needs to be done to merge those results together?

I mostly just split my indiced because it makes changes over time easier and because I didnt expect such a hit in terms of performance when querying multiple indices.

Should I try to put all my data into one big Index in order to avoid this kind of issues?

Another stange thing is, that I can not see any issues with regards to hot threads.
There is no thread which e.g. runs the same thing for the complete 'interval'.

Cheers,
Robert


(system) #2