I am considering CCR as a tool to migrate all the data (including historical data) from a source Elasticsearch single-node cluster to a target Elasticsearch multi-node cluster (both clusters have 7.17.7 version).
The existing cluster was created 3 years ago and now it has ~1TB of data and 20+ indices. All indices have index.soft_deletes.enabled = true.
After reading the documentation of CCR, it looks like it relies on shard history retention.
My question is that if I start creating follower indices now, will CCR replicate all the data from the source cluster, or only new data will be replicated that will be ingested after the follower indexes are created?
- If all data will be replicated, is there any chance that any of the records will be lost?
- if not, should I create new indices with
index.soft_deletes.retention_lease.period equals, for example, to 30/90 days, perform reindexing, and then use new indices as leader indices for CCR?
Thanks in advance!
Welcome to our community!
Why not do a remote reindex if you are looking to migrate?
The main issue with reindexing is that some of the indexes have over 1B of the documents, however the maintenance window is very short (max 4 hours). It also means that we will have to reindex all 20 indices in less than 4 hours, otherwise the application will not be able to operate.
My plan was to setup CCR replication and when all the data will be replicated, perform the cutover.
Are all 20 indices continously indexed into?
Yes, we are using Elasticsearch as a search engine for application data.
Also, do you have a platinum license for both of your clusters? CCR needs a paid license in both clusters.
Yes, we will have a license on both clusters, however, I hope that the replication will not take more than 1 month, so we will be able to decommission the old cluster quickly after that.
@leandrojmp, @Christian_Dahlqvist, @warkolm, Do you know if this approach is going to work at all or do I need to start looking into other options (reindexing or custom solutions)?
In theory it should work, the only way to know is by testing it.
I do not use CCR, so not sure what may happen.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.