Reindexing production environment handling live updates?

Elastic Stack Elasticsearch
1

Hi ,

I have gone through link for reindexing with zero downtime.

There is no information for handling live updates going on into old index.

Solution thought were:

  1. Queue updated and deletes and update new index with these instruction when reindexing is done.
    (Issue i can see with this as it does not update old index, my current search queries will not be up to date).

  2. Keep performing live updates on old index as well as keep queueing. When i am done with reindex, reissue queues command to new index.
    (Issue in this , there can be data inconsistencies).

  3. I can't use old index for reindex into new index as my old document did not contain some new fields. I always will be needing to reindex from source of truth (sql). Again as this sql db is getting updated at a high rate, how i can reindex to new elasticsearch index?

It will be really helpful if i can get some pointers.

Thanks in advance.

1 Like
2

Hi @Prannoy_Mittal,

you mention that your source of truth is your SQL database. In that case I'd duplicate the writes during this process and write the document into both indices if that is feasible.

This community member blog post about zero downtime reindexing also discusses a few options.

I hope that helps.

Daniel

3

thanks @danielmitterdorfer...the link is really useful. I think writing into old index and buffering the updates for newer index suits me the best.
Will explore more, don't you think writing to both index and also re indexing old data will lead can lead to inconsistent data with version conflicts?

4

Hi @Prannoy_Mittal,

you are right. The last part of the article describes some options for this case.

Daniel