# Reindex does not create all documents

**URL:** <https://discuss.elastic.co/t/reindex-does-not-create-all-documents/121368>\
**Category:** Elasticsearch\
**Created:** [February 25, 2018, 9:11am UTC](https://discuss.elastic.co/t/reindex-does-not-create-all-documents/121368 "2018-02-25T09:11:18Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dan\_Markhasin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_markhasin/32/14187_2.png) [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\
**Post date:** [February 25, 2018, 9:11am UTC](https://discuss.elastic.co/t/reindex-does-not-create-all-documents/121368/1 "2018-02-25T09:11:18Z")

</div>

Can anyone explain what is going on here? Why is the "created" count lower than "total"?

This is on ElasticSearch 5.4.0.

{  
"completed": true,  
"task": {  
"node": "6XQAqH3qQYqJq9MqCgpeLA",  
"id": 160658656,  
"type": "transport",  
"action": "indices:data/write/reindex",  
"status": {  
"total": 1268925,  
"updated": 0,  
"created": 1267357,  
"deleted": 0,  
"batches": 1268,  
"version\_conflicts": 0,  
"noops": 0,  
"retries": {  
"bulk": 0,  
"search": 0  
},  
"throttled\_millis": 0,  
"requests\_per\_second": -1,  
"throttled\_until\_millis": 0  
},  
"description": "reindex from [ipg.ipg\_fc\_timing\_indicator.pipeline.05810d74-0a06-46f4-b9eb-05f79f10a09d-latest] to [temp-ipg.ipg\_fc\_timing\_indicator.pipeline.05810d74-0a06-46f4-b9eb-05f79f10a09d-latest]",  
"start\_time\_in\_millis": 1519549530514,  
"running\_time\_in\_nanos": 118026405862,  
"cancellable": true  
},  
"response": {  
"took": 118026,  
"timed\_out": false,  
"total": 1268925,  
"updated": 0,  
"created": 1267357,  
"deleted": 0,  
"batches": 1268,  
"version\_conflicts": 0,  
"noops": 0,  
"retries": {  
"bulk": 0,  
"search": 0  
},  
"throttled\_millis": 0,  
"requests\_per\_second": -1,  
"throttled\_until\_millis": 0,  
"failures": []  
}  
}

---

<div class="post-metadata">

**Author:** ![Dan\_Markhasin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_markhasin/32/14187_2.png) [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\
**Post date:** [February 26, 2018, 7:59pm UTC](https://discuss.elastic.co/t/reindex-does-not-create-all-documents/121368/2 "2018-02-26T19:59:45Z")

</div>

Anyone? 😑

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [February 27, 2018, 1:41pm UTC](https://discuss.elastic.co/t/reindex-does-not-create-all-documents/121368/3 "2018-02-27T13:41:58Z")

</div>

I'm really not sure! The `total` is the total from the search responses for the scroll. The `created`, `updated`, `deleted`, and `noops` are counted based on how we process the results. I'm certainly curious which documents are missing. Can you check it? You'd have to write a program to walk the two indices in the same order and compare. What was the \_reindex request? Did you use ingest pipelines? It feels like a bug though.

---

<div class="post-metadata">

**Author:** ![Dan\_Markhasin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_markhasin/32/14187_2.png) [@Dan\_Markhasin](https://discuss.elastic.co/u/Dan_Markhasin)\
**Post date:** [February 27, 2018, 3:23pm UTC](https://discuss.elastic.co/t/reindex-does-not-create-all-documents/121368/4 "2018-02-27T15:23:32Z")

</div>

This is actually a continuation of this issue:

> [@Scroll returns inconsistent number of results](https://discuss.elastic.co/t/scroll-returns-inconsistent-number-of-results/118018):
>
> We are using ElasticSearch 5.4.0 and noticed a strange behavior of the scroll API on a very specific index. The behavior we are seeing is that when scrolling with a simple match\_all query, the number of returned results is sometimes different than the "total" returned by ES. Sometimes it returns the exactly expected number, but sometimes it returns less. It is happening on a specific index, that is no longer active (i.e. no new data is written to it) - we haven't seen this problem on any other…

It happened again, on a different cluster. Then, when I tried to reindex the data from the "faulty" index, I noticed that reindex itself created less documents than the "total" of the original index reported, so I don't think it is possible to retrieve the "missing" documents, as Elasticsearch simply doesn't return them - only counts them in the "total" for search/scroll requests.

The index request was a straight-forward reindex, with only source and dest specified, no pipelines were used.

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [February 28, 2018, 3:14pm UTC](https://discuss.elastic.co/t/reindex-does-not-create-all-documents/121368/5 "2018-02-28T15:14:19Z")

</div>

I've reached out to some folks who know more about the inner workings of scroll then I do to have a look at this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 28, 2018, 3:14pm UTC](https://discuss.elastic.co/t/reindex-does-not-create-all-documents/121368/6 "2018-03-28T15:14:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.