Transferring a writable index from one cluster to another

[Aditya_Teltia]

I am trying to migrate writable indices from one cluster to another. I wanted to know what could possibly the best approach for doing this.
Currently I am doing it as follows :

1. Phase1 -> Before taking snapshot of initial state redirect writes using ingest pipeline to a new index in source cluster and then take snapshot and restore it in destination index
2. Phase2 -> Shift read/write operation to new index in cluster2
3. Phase3 -> Take snapshot of new index and restore it to a new index in destination cluster
4. Phase4 -> Set the alias of both the indices in destination same
5. Phase5 -> Downgrade the source Index and new index in source cluster

I want to achieve this with zero downtime. I am okay with achieving eventual consistency.
Please help me with this asap.
Thank you,
Regards.

[Christian_Dahlqvist]

Why are you looking to do this? Is it a migration to a new cluster or something you will need to be doing on a regular basis? What exactly are the requirements?

[Aditya_Teltia]

"Actually, I am doing an internship and this is the desired task they want me to automate efficiently. Then I will integrate this automation of migration with their internal elastic cluster."

Yes, this is something that needs to be done on a regular basis as per the requirements I have received thus it needs to be efficient and there should not be any data loss.
I just want to know if there is any better approach to do this as I have tried reading a lot of articles regarding the same but haven't found anything reasonable.

[Christian_Dahlqvist]

Is your data immutable, e.g. there are no updates or deletes?

Why do you want to switch writing to the new cluster?

@Aditya_Teltia

Phase1 -> Before taking snapshot of initial state redirect writes using ingest pipeline to a new index in source cluster and then take snapshot and restore it in destination index

So you start indexing into a new index and then, once no more data is written to the old index, you snapshot it and restore it in the other cluster?

@Aditya_Teltia

Phase2 -> Shift read/write operation to new index in cluster2

This will create a new empty index that is being written to.

@Aditya_Teltia

Phase3 -> Take snapshot of new index and restore it to a new index in destination cluster

Once no more writes are hitting the new index in the source cluster you snapshot it and restore it as a new index in the destination cluster. Note that this can not be merged with the new index in the destination cluster without reindexing.

@Aditya_Teltia

Phase4 -> Set the alias of both the indices in destination same

I assume this is an alias you will use for querying and that it will need to point to all 3 indices.

@Aditya_Teltia

Phase5 -> Downgrade the source Index and new index in source cluster

I have no idea what downgrade means.

Would it not be easier to instead start writing to a new index in the destination cluster and then snapshot and restore the index no longer written to from the source to the destination cluster?

[Aditya_Teltia]

@Christian_Dahlqvist

Is your data immutable, e.g. there are no updates or deletes?

No, the catch is that the data is mutable. This is creating issues.

@Christian_Dahlqvist

So you start indexing into a new index and then, once no more data is written to the old index, you snapshot it and restore it in the other cluster?

I am doing this to separate the documents adding or getting updating during the snapshot and restore are going on so that I can this in the second pass of snapshot and restore.

@Christian_Dahlqvist

This will create a new empty cluster that is being written to.

This will be done from client side. We can skip this part as the client would be aware which cluster he have to query.

@Christian_Dahlqvist

Once no more writes are hitting the new index in the source cluster you snapshot it and restore it as a new index in the destination cluster. Note that this can not be merged with the new index in the destination cluster without reindexing.

No, since I have shifted the all the read and writes to destination cluster just after the first snapshot and restore. Now new index would not be having anymore queries to process and thus in the next phase I will snapshot it and restore it to a new index.

@Christian_Dahlqvist

I assume this is an alias you will use for querying and that it will need to point to all 3 indices.

Not 3 indices, the two indices in the destination cluster to where I have restored two of my source indices to.

@Christian_Dahlqvist

I have no idea waht downgrade means.

By downgrade I mean deleting the cluster1 indices as all the queries and data has been migrated to cluster2 [destination cluster] indices.

@Christian_Dahlqvist

Would it not be easier to instead start writing to a new index in the destination cluster and then snapshot and restore the index no longer written to from the source to the destination cluster?

That would make the system inconsistent for longer period of time since if the index have TBs of data it can take long to be restored that is why I was doing this in second pass since the data we will be migrating in second pass would be less.

Let me make the problem statement more clear

• I have a writable index in cluster 1
• I want to transfer it to cluster 2

How will you approach this ? Data is mutable and consider making the system inconsistent for minimum amount of time and downtime of the system should be zero.

[Christian_Dahlqvist]

@Aditya_Teltia

How will you approach this ? Data is mutable and consider making the system inconsistent for minimum amount of time and downtime of the system should be zero.

If the data is mutable I believe downtime will be required while you copy the index over. If you switch to new indices while copying the old one as you described, updates will result in duplicates or simply fail as no document to update is found. Sorting out duplicates with very large indices is very tricky. Also note that you can only perform indexing/updates/deletes against an alias if it only points to a single index. Once it points to multiple indices it goes read-only.

[leandrojmp (Leandro Pereira)]

@Aditya_Teltia

Let me make the problem statement more clear

• I have a writable index in cluster 1
• I want to transfer it to cluster 2

How will you approach this ? Data is mutable and consider making the system inconsistent for minimum amount of time and downtime of the system should be zero.

I had a similar need a couple of time ago and this will depend on how you are indexing your data.

What I did was:

• Start writing on both clusters a the same time using a different index name, if the original index name was my-index-name-2023.07, the new index name would be my-index-name-migration-2023.07 on both clusters, the queries would be done using an alias.
• Make a snapshot of the old indice in the first cluster, since this indice is not receiving new writes the data is now immutable.
• Restore this snapshot in the second cluster.

Which is not clear is this:

@Aditya_Teltia

Yes, this is something that needs to be done on a regular basis as per the requirements I have received thus it needs to be efficient and there should not be any data loss.

You will need to keep moving data from one cluster to another? Or you will just migrate once?

[Christian_Dahlqvist]

@Aditya_Teltia

Not 3 indices, the two indices in the destination cluster to where I have restored two of my source indices to.

The restore process can not merge indices so you would have the large one you stopped writing to, the new index you created for writing in the source cluster and the new write index you created in the destination cluster.

@leandrojmp I do not see how your approach works given that as far as I understand they have a single large index with mutable data that is continuously updated.

@Aditya_Teltia Exactly how you perform updates may also impact how complex it is. Is an update just replacing the full document with a new one or do you modify the existing documents, e.g. using partial or scripted updates?

[Aditya_Teltia]

@leandrojmp

You will need to keep moving data from one cluster to another? Or you will just migrate once?

By this I meant that this would be done on a regular basis with set of indices in the system. Not keep moving data from one cluster to another. I have to migrate it once.

@leandrojmp

Start writing on both clusters a the same time using a different index name

Say I get a query to update a document. How would this cater in second cluster as at this point that document is not present in second cluster ?

@leandrojmp

Restore this snapshot in the second cluster.

I will have to restore this snapshot to a new index (say my-index-name-migration-restored-2023.07). Since the snapshot cannot be restored to an open index right ? Now should I create an alias in second cluster as well for both indices (my-index-name-migration-2023.07 & my-index-name-migration-restored-2023.07).

Hopefully my doubts are making sense, as I have been using and studying the architecture of Elasticsearch for the past 5 weeks, so I am quite a rookie in this tech. Let me know if something is still unclear, I will try to explain better.

[Aditya_Teltia]

@Christian_Dahlqvist

Exactly how you perform updates may also impact how complex it is. Is an update just replacing the full document with a new one or do you modify the existing documents, e.g. using partial or scripted updates?

Modify the existing document using partial or scripted updates and even can delete the index.
In case of deleting, I am not sure how can I migrate this change to cluster2 index.

Is there some way I can store logs of set of queries coming to an index for a specific time period and then roll these changes ( run these queries ) to the index in cluster2. What I mean is.

• I take inital snapshot and restore it to cluster2.
• While this restore is ongoing, there might be some updates happening to the index in cluster1. Is there a way, I can store the logs of these updates and eventually roll them after restoration process is completed ?

[Christian_Dahlqvist]

@Aditya_Teltia

While this restore is ongoing, there might be some updates happening to the index in cluster1. Is there a way, I can store the logs of these updates and eventually roll them after restoration process is completed ?

You would have to build that yourself. Also note that the order of operations may matter, so you may need to keep buffering operations until you have finished replaying them.

I see no good way to do this so would like to know what the reason for moving data between clusters is. Knowing the rationale behind the request may make it easier to come up with alternate solutions.

Are the clusters in close proximity?

[Aditya_Teltia]

@Christian_Dahlqvist

You would have to build that yourself. Also note that the order of operations may matter, so you may need to keep buffering operations until you have finished replaying them.

Do you have any suggestions, how can I approach this ?

@Christian_Dahlqvist

I see no good way to do this so would like to know what the reason for moving data between clusters is. Knowing the rationale behind the request may make it easier to come up with alternate solutions.

Are the clusters in close proximity?

I personally have asked about the reason from my manager, will get back once I am well versed with the reason. I should have got it cleared prior sorry. Surely knowing the rationale would make it easier.

Also, thank you for being so responsive. I will get back once I understand the reason clearly.

[Aditya_Teltia]

@Christian_Dahlqvist

Knowing the rationale behind the request may make it easier to come up with alternate solutions.

There are mainly three reasons of doing this as I understood -

1. If the available space in cluster is less, we have to migrate the index with zero downtime.
2. There are many clusters of different teams in our firm [say team1, team2], sometimes we tend to migrate indices which are using resources of a team say team1 but not actually useful for team1.
3. Changing cluster in general.

Now based on these can you think of some good approach for the same.

[Christian_Dahlqvist]

@Aditya_Teltia

Now based on these can you think of some good approach for the same.

As I said earlier I do not think it is possible to move an index that is receiving updates/deletes from one cluster to another without data loss, inconsistencies and/or unavailability, at least using snapshot and restore.

The only way I can see to achieve this with a minimum amount of downtime is to use cross-cluster replication (CCR), which is a commercial feature.

With CCR you would create a follower index in the destination cluster and then wait until it has caught up with writes/updates/deletes on the main source index. At that point you can temporarily stop indexing, stop replication and make the follower index writable. You can the start indexing into the new index and delete the one in the source cluster. As the index already is up to date this would ensure consistency with likely only a small amount of downtime.

[Aditya_Teltia]

@Christian_Dahlqvist

As the index already is up to date this would ensure consistency with likely only a small amount of downtime.

As I mentioned earlier as well, our system is ok with eventual consistency but we cannot afford downtime. That is why I am currently using a two pass approach which is as follows:

Phase 0: Pre Migration checks

pre_migration_check(dest_client, source_client, source_index)
Checking whether cluster2 has enough disk space to store source_index or not.

Phase 1: Change write pipeline

change_pipeline(source_client, pipeline_name, new_index, source_index)
Setting _ingest pipeline for source_index to field: _index, value: new_index, so that all the updates and writes from now on for source_index go to new_index. Doing this to segregate writes from this point in time.

Phase 2: Snapshot and restore [First Pass]

snaps(source_client, source_repository, source_index, dest_client, dest_index)
Taking snapshot and restoring it to the dest_index.

Phase 2.1: Verifying snapshot and restore

verifying_snapshot_and_restore(source_client, source_index, dest_client, dest_index)
Verifying by comparing the count of documents in source_index and dest_index.

Making pipeline null in destination index

make_null(dest_client , dest_index)
Setting default pipeline to null in dest_index as it has also been restored from source_index.

---- Shift operations from source_index to dest_index from client side.----

Phase 3: Reindexing new index [2nd Pass]

reindex_data(source_host, dest_client, new_index, dest_index)
Reindexing the new_index having the delta documents that got added while the first snapshot and restore was running, so that the dest_index becomes up to date.

Phase 3.1: Post second pass verification

second_pass_verification(source_client, source_index, new_index, dest_client, dest_index)
Verifying by comparing the count of documents in source_index + new_index and dest_index.

Phase 4: Cleanup

delete_index(source_client , source_index)
delete_index(source_client , new_index)

Deleting or downgrading indices in cluster1 as now they are irrelevant.

Obviously there will be some redundant documents in the process because this whole process cannot incorporate deletes on the main source_index. Let me know if you see any other flaws in this.

[Christian_Dahlqvist]

I suspect the result of the process you describe will depend on how you perform and handle updates. Let us assume you have a document in the main index and perform scripted updates on this, e.g. to add or modify elements in a list within the document. This is just one example where I can see potential issues. You can probably walk through other examples and see how that fares.

Assume you start the process and that this due to the size of the index takes a fair bit of time. You create a new index and direct writes there while you perform snapshot/restore. While this is going on an update for the document comes in and as it does not exist in the index you may end up creating a new one with just new content. If something was to be deleted that would potentially just have been dropped.

Once the restore has been completed you point to the new index and start reindexing from the new index, which could be reasonably large for a busy index. While you are reindexing, another update for the document comes in. You now have 2 updates that need to be performed. Does the order matter? Will they be applied correctly given the partial document that was stored in the new index?

I think you have the potential to end up with inconsistent documents that will not have the structure that is expected. I do not see this as eventually consistent.