We have 7 data nodes and 90 indices with all having 1 shard except one. This particular index has 14 shards and the data received in a day is around 400Gb on the index. The indexing rate is too high on the index.
All the primary shards of the index(time(day) based rolling) are created on a single node. As the index rate is high plus the search is mostly done on this index, we observe a lot of out of memory issue. The ES does not recover on its own as well.
The nodes currently are on 32gb RAM with 15GB assigned to ES.