Hi,
I have a cluster of 9 nodes, 3 master-hot, 3warm, and 3 cold, they have the same amount of ram 32GB but different disk-space, the hot is on SSD, the others on the mechanic.
I haven't followed the ratio best practice like 1:30 for the hot, 1:160 warm, and 1:250 cold, but I do have enough space like 250TB across all the nodes.
Now I am ingesting something like 1TB ~ 1.3TB od data per day, quite a lot, and I thought my cluster was enough but it does not look like, coz I am having issues when it comes to search and so on.
I have allocated 18 primaries shards on hot and 6 on warm/cold using the default replica:1.
I really do not know what kind of action should I take in order to optimize my cluster, any help on this?
Shall I increase my replica in order to increase the performance search?
Shall I increase the number of shards? but I do not think so because most of the time I have "Waiting for all shard copies to be active" or another type of warning related to the shards.
That sounds like a very small cluster given the ingest volume and I would expect you to suffer from heap pressure which leads to performance problems. I would recommend you upgrade to Elasticsearch 7.7 which contains some changes that dramatically reduce heap usage.
If you for some reason are unable to upgrade I would recommend watching this webinar.
indeed about the upgrade but ain't gonna happen soon, I checked the webinar and they explain pretty much well the approach warm/hot/cold, the only difference is that I do not have dedicate master, I mean my master are hot nodes as well.
How can I check the volume of data that my cluster can ingest? it is small indeed, but I only noted now after almost 3 months of usage that it is small.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.