We're using ES cluster (version 5.4.1) with 4 data nodes, 3 master, one client node (kibana).
The data nodes are r4.2xlarge aws instance (61g memory, 8vCPU) with 30G memory allocated for the ES JAVA.
We have writing of around 200G of logs every day and keep it for the last 14 days.
A one big index of 160G-170G a day (6 shards, 1 replica) and other smaller indices of 1-3G (2 shards, 1 replica)
We're dealing with performance latency in the query duration and I'm looking for recommendations to our cluster to improve the cluster performance, especially the search performance - query duration (kibana).
For example, searching for the last 6 days on the big index takes:
Query Duration 51498ms
Request Duration 52706ms
More data nodes? more client nodes? bigger nodes? more replica's? maybe improve the queries duration at the expense of writes speed(?) - anything that can improve the performance is an option.
Is there anyone with something close to this design or loads? I'll be glad to hear about other designs, loads and query stats.