I got an elasticsearch cluster (6.4.2) with hundreds of monthly indices each of which has a varying size between 500 gigabytes and 1 terabyte consisting of 5 primary and 5 replica shards (default).
The question is;
When someone query some data ranging more than 2 years (24+ monthly indices) by the web application connected to this cluster, what i observe in the slowlogs in elasticsearch side is; same query is sent to each index that is between the specified time range, and the cluster starts to respond slower, garbage collection logs are printed and eventually nodes get OOM one by one.
Is there a solution to overcome this problem? Maybe a configuration about concurrency or memory management?
Thanks in advance.