How to reindex an index that is being actively modified

Elastic Stack Elasticsearch
1

Hello!

I need to reindex an index that is being actively modified (docs added and updated) and searched via an alias. What is the best way to do that while minimizing the data gap in the new index due to the old index being actively modified?

The problem I’m hitting is that when I do /reindex and point the index alias with the is_write_index: true to the new index, I start getting error about missing documents, such as:

Mapping wasn't updated for id 24826, reason [index-2024-10-15/oXSR7_1dQ5idMNMwlWGu3w][[index-2024-10-15][1]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][1300565]: document missing]], doc: update {[alias-1][_doc][24826], doc_as_upsert[false], doc[index {[null][_doc][null], source[{"lastSeenTimestamp":1729214166852}]}], scripted_upsert[false], detect_noop[true]}

Reindexing + alias command(s) I’m using:

curl -XPOST -H 'Content-Type: application/json' 'http://localhost:9200/_reindex?wait_for_completion=false' -d '{
   "source": {
    "index": "index-20190221"
   },
   "dest": {
      "index": "index-2024-10-15"
   }
}'

curl -H 'Content-type:application/json' -XPOST "localhost:9200/_aliases" -d '{
      "actions": [
        { 
          "remove" : {
              "index" : "index-20190221", 
              "aliases" : ["alias-1", "alias-2"]
          }
        },
        { 
          "add" : {
              "index" : "index-2024-10-15", 
              "aliases" : ["alias-1", "alias-2"],
              "is_write_index": true
          }
        }
      ]
}'

curl -XPOST -H 'Content-Type: application/json' 'http://localhost:9200/_reindex?wait_for_completion=true' -d '{
   "source":{
    "index":"index-20190221",
    "query": {
      "range": {
        "key": {
          "gte": "<lastTimestamp_in_new_index>"
        }
      }
    }
   },
   "dest":{
      "index":"index-2024-10-15"
   }
}'

If I do not include the is_write_index option for the new index, the index seems to be ignored - no new documents get added to it, but I guess that's how it supposed to be. Also, I’m defining custom IDs for index documents.

My reindexing plan is as follows:
0. create new index

  1. reindex
  2. after reindex is done successfully, check the latest document timestamp in new index
  3. switch alias (remove is_write_index from the old and add it for the new index)
  4. reindex all documents from the old index with timestamp newer than the one in step 2, to "fill the gaps"

Is this the recommended procedure? Or is there a better one to reindex with potentially zero data loss?

Thank you in advance,
Matt