I have a 5 node ES cluster and it's performing well on an average but sometime 0.5% of the queries are taking 1 sec to respond. I have tried all the possible solutions/suggestions, not sure what is causing issue.
Overview of cluster:
3 Master with 4gb allocated heap out of 8 gb RAM
2 data node with 4 gb allocated heap out of 64 gb RAM and 70 % of heap is utilized, should i consider increasing the heap size ( Average throughput is 15000 tps).
On all the master OS mem is 94 % used.
Please suggest what should i do to handle 15000 tps with no slow queries.