Performance impact of changing number_of_replicas on a live index

[ghallas]

Hello,

I have a cluster with 3 master nodes and 9 data nodes.

My main index has 3 shards, and number_of_replicas:1 . It also has some 5 million documents, and receives around 30 requests per second.

Data node CPU utilization is very low at 5%, except for 3 nodes that are effectively idle.

This is obviously due to the number of shards and replicas. The initial intention was to have number_of_replicas:2but I mistyped during index creation.

My question is: can I safely modify the number of replicas while the index is being queried and is ingesting? Should I expect performance degradation after I modify the number of replicas?
Additionally: when should I expect the replicas to be synced with the primary shards? Is the copying of data from the existing shards something that happens gradually in the background?

Thanx!

[spinscale (Alexander Reelsen)]

Hey,

you can safely change the number of replicas. The change to copy the index to more nodes happens transparently in the background while indexing can go on. What happens is, that a new copy is created on another node, which needs some network I/O and a little bit of CPU to keep things in sync, but given that you system does not seem to be overloaded, I would not worry.

--Alex