We have a 18 node elastic search cluster. We have a few indices but lets talk about one specific index. It has 60 shards with a replica of 1. There are about 11 million documents and the index size is about 850GB.
We are getting into a process of re-indexing regularly. So as part of re-indexing, among other settings, I have set the new index's replica=0. Lets say we complete the re-indexing operation in a few hours and once re-indexing completes, we will set replica=1.
Here are the questions I have:
- What happens after I increase replica of a newly created index from 0 -> 1.
- How long does it take ES to create the new replica shards for all these 60 shards for the new index?
- Does ES cluster health turn yellow when I increase the replica count? Alternatively, how does ES communicate to us that its replicating the shards?
- What should I monitor to know when all the 60 shards have replicated?
- Should I wait to switch the alias from the older index to the newer index till the replication process has completed? In other words, can the new index start taking live traffic (both indexing and search traffic) while the replica are still being setup?
From what I have read, indexing is done on the primaries while searching can be done on the replicas. So I guess its safe to continue to index but not sure about searching.
Please advise. I searched for answers but couldnt find anything relevant.
We are on v 6.3 of Elastic Search.