Cluster configuration, shards, and replica

Elastic Stack Elasticsearch
1

Hi,
I have a cluster of 9 nodes, 3 master-hot, 3warm, and 3 cold, they have the same amount of ram 32GB but different disk-space, the hot is on SSD, the others on the mechanic.
I haven't followed the ratio best practice like 1:30 for the hot, 1:160 warm, and 1:250 cold, but I do have enough space like 250TB across all the nodes.

Now I am ingesting something like 1TB ~ 1.3TB od data per day, quite a lot, and I thought my cluster was enough but it does not look like, coz I am having issues when it comes to search and so on.

I have allocated 18 primaries shards on hot and 6 on warm/cold using the default replica:1.

I really do not know what kind of action should I take in order to optimize my cluster, any help on this?
Shall I increase my replica in order to increase the performance search?
Shall I increase the number of shards? but I do not think so because most of the time I have "Waiting for all shard copies to be active" or another type of warning related to the shards.

Any help will be much appreciated.

2

That sounds like a very small cluster given the ingest volume and I would expect you to suffer from heap pressure which leads to performance problems. I would recommend you upgrade to Elasticsearch 7.7 which contains some changes that dramatically reduce heap usage.

If you for some reason are unable to upgrade I would recommend watching this webinar.

3

indeed about the upgrade but ain't gonna happen soon, I checked the webinar and they explain pretty much well the approach warm/hot/cold, the only difference is that I do not have dedicate master, I mean my master are hot nodes as well.

How can I check the volume of data that my cluster can ingest? it is small indeed, but I only noted now after almost 3 months of usage that it is small.

Thanks for the reply btw.

4

If you have monitoring installed I would recommend starting by looking at heap usage across your nodes and ensure this looks OK.