We are deploying a elasticsearch cluster, on kubernetes (using the official helm chart).
However, we have an unresolved issue. Our usage of elasticsearch has huge variations in usage. During night, we can have less than 1req/s and during day we might go up to 1000req/s.
My plan to solve this issue was to have 2 group of node, the primary node (with contains all the primarysshard + 1 replica) and an other group of node than can scale from 0 to X and contains only replica of the shard.
However, I still can not find the option to force a group of node to contains only replica shard, and never became primary.
Does anyone have idea?
One option I was thinking was to forbid primary shard reallocation, but sometimes, we need to create new index and it might be little annoying?
How does people solve that kind of problem?
Thx a lot