Strange reindex behaviour

[leandrojmp (Leandro Pereira)]

I'm doing some reindex of daily index to monthly ones and sometimes the reindex process enters in a loop and starts again after it already reindexed the documents.

For example, considering the index indexName-2021.05.01 with 500.000 documents and reindexing it into indexName-2021.05, I start the reindex process with the following request:

POST _reindex
{
  "source": {
    "index": "indexName-2021.05.01"
  },
  "dest": {
    "index": "indexName-2021.05"
  }
}

Then I use GET _tasks?actions=*reindex&detailed to get the task id and GET _tasks/taskID to monitor the progress.

I can see the created number increasing as expected.

          "status" : {
            "total" : 500000,
            "updated" : 0,
            "created" : 184000,
            "deleted" : 0,
            "batches" : 184,
            "version_conflicts" : 0,
            "noops" : 0,
            "retries" : {
              "bulk" : 0,
              "search" : 0
            }

I also use the Discover in Kibana to compare the documents, in one tab I filter by the daily index and in the other i filter by the monthly index, both using the same time interval, which correspond the the time interval present in the daily index.

After sometime I check again for the progress using the task id and got an answer saying that the task doesn't exist anymore, from discover I can see that all documents were reindexed as expected.

But if I run GET _tasks?actions=*reindex&detailed to get the tasks running I can see few tasks doing reindex for the same index, but now they show the updated part of the status response increasing.

          "status" : {
            "total" : 500000,
            "updated" : 175000,
            "created" : 0,
            "deleted" : 0,
            "batches" : 175,
            "version_conflicts" : 0,
            "noops" : 0,
            "retries" : {
              "bulk" : 0,
              "search" : 0
            }

My question is: What can cause this to happen since the documents were already indexed?

Sometimes this reindex loop takes a long time, which makes the load and CPU of the nodes doing the reindex process to increase a lot.

Normally when this happens and I can confirm that all the documentes were reindexed, I cancel all the reindex tasks using:

POST _tasks/_cancel?actions=*reindex

But I would like to know why this is happening and how to avoid it.

I'm currently running version 7.9.3, an upgrade to 7.12.1 is planned for the next weeks.