I'm running 3 node (26 GB EBS volume, 4GB Memory, c5.large) ES cluster v7.7 (AWS) for 1 index with the following setting
PUT /my-index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1
}
}
After initial research, I understood that i will not get the shard allocation "right", unless at least i know ahead of time the size of index etc. Therefore i believe now it's the time to re evaluate the setup.
This index currently holds searchable ~37M documents (deleted documents currently sit at ~4M) and its occupying 5GB. It is not expected to grow rapidly. By end of the year might increase to 40M docs.
What id like to understand is what I got wrong as the cluster is experiencing the following:
- One node not receiving search requests (screenshot attached)
- At random points in time, the cluster is having spikes in Search Latency over 6secs, http 400 response codes are thrown, without any cpu spike indication, and the way its "resolved", its by not sending any more requests to the ES cluster.