We currently have 6 ES data nodes, 1 ES master node, and 1 ES coordinating node in our setup. We are conducting load tests that generate millions of lines of data, which get fed from hundreds of server using Filebeat that directs the data to our 6 Logstash nodes and then eventually to our ES cluster.
the structure of the data is timeseries. We collect metrics such as response time, response codes, latency, etc..
Over the course of 2-3 hours our index size are 200+ gb and doc count are over 200million
Specs for each node:
Each index has 192 shards and each index represents 1 test.
ES data node - AWS m5.4xlarge (4 core , 16gb ram with 1tb of ssd mem)
ES coordinating node - AWS r5.2xlarge ( 8 core, 64 gb ram with 1 tb of ssd mem)
ES master node - AWS r5.2xlarge ( 8 core, 64 gb ram with 1 tb of ssd mem)
We are trying to get real-time data as the data is being ingested into ES, but eventually, the ES queries are taking longer as more data is coming in.
My question is, what is the best way to configure our ES and what are some steps to figure out the correct shard count for each index.