We have a 3 node cluster with 128GB RAM in each node. Heap allocated for each node is 31GB. Our expected monthly record count is 8 billion per month. So we have created daily indices to hold data up to 2 years. Now, after 9 months heap usage always stays around 80% and sometimes all nodes crash at the same time due to OutOfMemory exception. Can the heap usage reduced by reindexing the documents with monthly indices? If so, what kind of heap usage improvement can I expect?
Is there any other way I can reduce heap usage without reducing number of shards?