# Not all documents copied after reindex

**URL:** <https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235>\
**Category:** Elasticsearch\
**Created:** [July 17, 2018, 12:42am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235 "2018-07-17T00:42:53Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 17, 2018, 12:42am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/1 "2018-07-17T00:42:53Z")

</div>

I modified a template for the index to convert a couple of variables from string to long.  
Then I issued a reindex call:

```
POST _reindex
{
  "source": {
    "index": "index-old"
  },
  "dest": {
    "index": "index-new"
  }
}

```

No errors anywhere, but I'm missing about half the documents when the task completes:

```
get _cat/indices/index-*
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open index-old tEdSeRmFQ7WhGFoWb4vYVw 6 1 993613 0 1.9gb 986.7mb
green open index-new 3KOMWY9lSyyASJS9TgPMEQ 6 1 532999 0 1gb 541.2mb    

```

I've done this many times before and reindex is usually very reliable. How can I figure out what happened to the rest of the documents?

Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 17, 2018, 12:47am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/2 "2018-07-17T00:47:19Z")

</div>

Is there anything in your Elasticsearch logs that might explain it?

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 17, 2018, 1:04am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/3 "2018-07-17T01:04:02Z")

</div>

Nothing. I was tailing the logs in real time on the server where this task was running. Not a single entry.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 17, 2018, 1:15am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/4 "2018-07-17T01:15:46Z")

</div>

Very odd, can you post the mappings from each index and a sample doc?

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 17, 2018, 1:42am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/5 "2018-07-17T01:42:59Z")

</div>

Mappings are ~800 lines long. Do you want just the differences or the whole thing?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 17, 2018, 2:33am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/6 "2018-07-17T02:33:43Z")

</div>

What about putting it into gist/pastebin/etc?

---

<div class="post-metadata">

**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\
**Post date:** [July 17, 2018, 3:43am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/7 "2018-07-17T03:43:27Z")

</div>

reindex not complete？  
see the tasks show the information  
/\_cat/tasks?pretty&v

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 17, 2018, 4:27am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/8 "2018-07-17T04:27:31Z")

</div>

Reindexing task definitely completes. I monitor it in the log files and with

```
GET _tasks?detailed=true&actions=*reindex
```

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 17, 2018, 4:35am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/9 "2018-07-17T04:35:58Z")

</div>

Old mapping:  
[https://pastebin.com/fxnsHE27](https://pastebin.com/fxnsHE27)

New mapping:  
[https://pastebin.com/8dGyEgFe](https://pastebin.com/8dGyEgFe)

Example doc:  
[https://pastebin.com/ZTQLqxTS](https://pastebin.com/ZTQLqxTS)

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 18, 2018, 12:59am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/10 "2018-07-18T00:59:42Z")

</div>

Another index is now exhibiting identical problem. Copies about half the data and just stops without errors. This one is much larger so I don't think its a resource problem.

Any ideas?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 18, 2018, 1:09am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/11 "2018-07-18T01:09:38Z")

</div>

What version are you on?

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 18, 2018, 2:39am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/12 "2018-07-18T02:39:32Z")

</div>

elasticsearch-6.2.3

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 19, 2018, 12:39am UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/13 "2018-07-19T00:39:55Z")

</div>

Couple of other things I noticed:  
It doesn't always stop on the same document.  
Out of total of 1151419 documents, first run quit on 634999 and second on 651999.

I was also looking at \_nodes/stats and didn't see any significant differences in nodes running the reindex tasks and those that weren't.

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 20, 2018, 4:35pm UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/14 "2018-07-20T16:35:51Z")

</div>

I tried reindexing into a different name to make sure errors in the template aren't affecting reindexing and the same problem exists. Only about half the data got copied.

Does anyone have any other suggestions for troubleshooting/debugging this?

Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 20, 2018, 9:30pm UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/15 "2018-07-20T21:30:22Z")

</div>

It might be worth raising an issue on GitHub.

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 20, 2018, 9:48pm UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/16 "2018-07-20T21:48:33Z")

</div>

Slightly unrelated question:  
What happens when template includes a mapping for a variable to be cast as long, but variable comes in with quotes like "1024"? Does it get discarded? I noticed that the sum docs.count + docs.deleted are close to the total number of documents in the original index. So while there are still some documents completely missing from destination index this would at least explain part of it.

In my latest test out of 1151419 total documents, destination index contains docs.count = 684045 and docs.deleted = 196412. So while 270962 are still missing, at least 200k are simply deleted.

I also tried reindexing into a name that doesn't match any templates and all documents made it across. So while it might still be a bug worth reporting on GitHub I suspect I just don't have a full understanding of how mapping changes affect reindexing.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 22, 2018, 8:42pm UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/17 "2018-07-22T20:42:20Z")

</div>

Deleted docs could be an indicator of things being overwritten.

Are you using beats as the data source? Are they passed to anything else before they hit Elasticsearch?

---

<div class="post-metadata">

**Author:** ![daq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daq/32/33426_2.png) [@daq](https://discuss.elastic.co/u/daq)\
**Post date:** [July 23, 2018, 5:09pm UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/18 "2018-07-23T17:09:42Z")

</div>

Deleted docs only show up during reindexing, but original source is filebeat.  
Full path for data is filebeat -\> logstash (all filtering happens here) -\> redis -\> logstash -\> elasticsearch.

I'm not altering the data in any way during reindexing. Only the mapping template is different since I'm trying to remap a couple of strings into longs.

Do I need to do it by casting the variable in a script or should updating mapping in a template enough?

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 20, 2018, 5:09pm UTC](https://discuss.elastic.co/t/not-all-documents-copied-after-reindex/140235/19 "2018-08-20T17:09:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.