Best way to reindex very active index

Elastic Stack Elasticsearch
1

When migrating to a new index, what's the best way to handle ensuring there's no differential between the two indexes and migrating aliases for zero downtime.

So let's say we are adding 10k documents per second in an index INDEX_A with an alias INDEX and we reindex that to INDEX_B. As soon as that re-indexing is complete let's say we update the alias, there's potentially a few thousand documents added to INDEX_A after reindexing. Is there a way to ensure that once reindexing is complete that any new documents are added to both or what's the idiomatic way to handle reindexing an active index?

2

At the moment what we're doing is, from the application side, duplicating indexing to both the old and new index, then reindexing, and once reindexing is complete, from the application side, only sending documents to the second index, and shifting searches to the second index. I just wanted to be sure there wasn't a better way to handle this in an atomic way within ES.

3

That is the best way! :slight_smile:

4

i am interested to see how you do it on with the application side, would appreciate the details

i usually reindex only to change mappings for that:
A) i stop logstash so new data do my thing with the active index and start it
B) update old indexes and the template dont touch todays index, tomorrow it will write to new index with new mapping and i move the old one (my indexes are daily)
C) update logstash index name, restart move old ones to new index :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.