I have a cluster of 9 nodes, 3 master-hot, 3warm, and 3 cold, they have the same amount of ram 32GB but different disk-space, the hot is on SSD, the others on the mechanic.
I haven't followed the ratio best practice like 1:30 for the hot, 1:160 warm, and 1:250 cold, but I do have enough space like 250TB across all the nodes.
Now I am ingesting something like 1TB ~ 1.3TB od data per day, quite a lot, and I thought my cluster was enough but it does not look like, coz I am having issues when it comes to search and so on.
I have allocated 18 primaries shards on hot and 6 on warm/cold using the default replica:1.
I really do not know what kind of action should I take in order to optimize my cluster, any help on this?
Shall I increase my replica in order to increase the performance search?
Shall I increase the number of shards? but I do not think so because most of the time I have "Waiting for all shard copies to be active" or another type of warning related to the shards.
Any help will be much appreciated.