Reindex does not create all documents

[Dan_Markhasin]

Can anyone explain what is going on here? Why is the "created" count lower than "total"?

This is on ElasticSearch 5.4.0.

{
"completed": true,
"task": {
"node": "6XQAqH3qQYqJq9MqCgpeLA",
"id": 160658656,
"type": "transport",
"action": "indices:data/write/reindex",
"status": {
"total": 1268925,
"updated": 0,
"created": 1267357,
"deleted": 0,
"batches": 1268,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0
},
"description": "reindex from [ipg.ipg_fc_timing_indicator.pipeline.05810d74-0a06-46f4-b9eb-05f79f10a09d-latest] to [temp-ipg.ipg_fc_timing_indicator.pipeline.05810d74-0a06-46f4-b9eb-05f79f10a09d-latest]",
"start_time_in_millis": 1519549530514,
"running_time_in_nanos": 118026405862,
"cancellable": true
},
"response": {
"took": 118026,
"timed_out": false,
"total": 1268925,
"updated": 0,
"created": 1267357,
"deleted": 0,
"batches": 1268,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0,
"failures": []
}
}

[Dan_Markhasin]

Anyone?

[nik9000 (Nik Everett)]

I'm really not sure! The total is the total from the search responses for the scroll. The created, updated, deleted, and noops are counted based on how we process the results. I'm certainly curious which documents are missing. Can you check it? You'd have to write a program to walk the two indices in the same order and compare. What was the _reindex request? Did you use ingest pipelines? It feels like a bug though.

[Dan_Markhasin]

This is actually a continuation of this issue:

Scroll returns inconsistent number of results

We are using ElasticSearch 5.4.0 and noticed a strange behavior of the scroll API on a very specific index. The behavior we are seeing is that when scrolling with a simple match_all query, the number of returned results is sometimes different than the "total" returned by ES. Sometimes it returns the exactly expected number, but sometimes it returns less. It is happening on a specific index, that is no longer active (i.e. no new data is written to it) - we haven't seen this problem on any other…

It happened again, on a different cluster. Then, when I tried to reindex the data from the "faulty" index, I noticed that reindex itself created less documents than the "total" of the original index reported, so I don't think it is possible to retrieve the "missing" documents, as Elasticsearch simply doesn't return them - only counts them in the "total" for search/scroll requests.

The index request was a straight-forward reindex, with only source and dest specified, no pipelines were used.

[nik9000 (Nik Everett)]

I've reached out to some folks who know more about the inner workings of scroll then I do to have a look at this.