I have a cluster with 3 master nodes and 9 data nodes.
My main index has 3 shards, and
number_of_replicas:1 . It also has some 5 million documents, and receives around 30 requests per second.
Data node CPU utilization is very low at 5%, except for 3 nodes that are effectively idle.
This is obviously due to the number of shards and replicas. The initial intention was to have
number_of_replicas:2but I mistyped during index creation.
My question is: can I safely modify the number of replicas while the index is being queried and is ingesting? Should I expect performance degradation after I modify the number of replicas?
Additionally: when should I expect the replicas to be synced with the primary shards? Is the copying of data from the existing shards something that happens gradually in the background?