How to migrate index A to index B in Elasticsearch without stopping writes to index A?

Elastic Search
1

I'm using Elasticsearch 7.10.0 and need to migrate data from index A to a new index B without interrupting write operations to index A. I want to achieve zero downtime during this migration and to migrate both the existing (historical) data and the new, incoming (incremental) data.

Here's my current situation:

  • Elasticsearch Version: 7.10.0

  • Source Index: A (actively receiving writes)

  • Destination Index: B (newly created, needs to be populated with data from A)

  • Constraint: I've found that using the Reindex API in my 7.10.0 setup can lead to significant performance degradation on index A or even data inconsistency issues. Therefore, I'm looking for alternative solutions that avoid or minimize the use of Reindex for bulk data transfer.

  • Requirement: Migrate both the full historical data from A and ensure that all new data written to A after the initial migration starts is also continuously synced to B.

Any advice, examples, or pointers to relevant documentation would be greatly appreciated. Thanks in advance for your help!

2

This is really old and have been EOL and without security updates for a long time. I recommend that you upgrade to the latest version of Elasticsearch.

Are you just adding new data or also performing updates and/or deletes?

3

I get it, this version is super old , but we’re stuck with it for now because of some external reasons. My data is just write-only though—no updates or deletes.
Any idea how I can fix this? Would really appreciate you help!

4

That makes it a lot easier. As you will need to reindex all the data there will be additional load and there is no way to avoid that. I would probebly try creating the new index and then redirecting new writes there. Once the old index is no longer written to you can start a reindexing job of this data to the new index. You will have duplicates for a period of time if both indices are included in queries, but this will go away once reindexing is done and the old index is deleted.

5

Thank you this is very helpful to me.

I will design a technical solution based on your ideas (ﾉ>ω<)ﾉ