# Count of docs after \_reindex'ing higher than before

**URL:** <https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015>\
**Category:** Elasticsearch\
**Created:** [January 18, 2018, 10:19am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015 "2018-01-18T10:19:39Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 1:32pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/21 "2018-01-18T13:32:09Z")

</div>

Nope. Nothing is found. Houdini document 😛

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [January 18, 2018, 1:33pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/22 "2018-01-18T13:33:40Z")

</div>

Then, the only way I see this could happen is if some process somewhere indexes a new document in the new index. Is that possible?

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 1:38pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/23 "2018-01-18T13:38:48Z")

</div>

We have a process that runs every evening but it updates data for the last month only. It definitely wouldn't be looking at documents that are years old.

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [January 18, 2018, 1:45pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/24 "2018-01-18T13:45:42Z")

</div>

You're not using an alias as the source index name, but are you using a wildcard in the name by any chance (it wasn't clear from your example above)?

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 1:48pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/25 "2018-01-18T13:48:17Z")

</div>

Nope. All indices follow the pattern `index_name-year.month` but they are explicitly provided, e.g. `orders_prod-2018.01`, without any wildcards.

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [January 18, 2018, 1:54pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/26 "2018-01-18T13:54:14Z")

</div>

And of course, dump question, but your destination index is completely empty (or even not existing) before you kick off the reindex call?

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 1:55pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/27 "2018-01-18T13:55:40Z")

</div>

The index is created just before making the call, so yes - before the call it is completely empty.

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [January 18, 2018, 1:56pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/28 "2018-01-18T13:56:28Z")

</div>

Do you mind sharing the response you get from the reindex call?

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 2:04pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/29 "2018-01-18T14:04:51Z")

</div>

I often (so far always when reindexing orders) get

> {  
> "statusCode": 504,  
> "error": "Gateway Time-out",  
> "message": "Client request timeout"  
> }  
> which is something I didn't find the answer for - but the docs still got indexed so I totally forgot about that little fact.

I'm trying to get the example of a non-timing out example...

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [January 18, 2018, 2:06pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/30 "2018-01-18T14:06:58Z")

</div>

Note that you can always get the status of an ongoing reindex task even if the browser has timed out by running this:

```
GET _tasks?detailed=true&actions=*reindex

```

And I would also add `wait_for_completion=false` in your reindex call, so that you can query the task status even after the reindex task has finished. And if you do that, you'll need to make sure to delete the task document manually once you've finished inspecting it, e.g.

```
DELETE /_tasks/taskId:r1A2WoRbTwKZ516z6NEs5A:36619
```

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 2:39pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/31 "2018-01-18T14:39:27Z")

</div>

Hmmm... When the task was running I kept an eye out on its output and there was nothing indicating any issues. After the task completed I couldn't retrieve its details though... Getting

> {  
> "nodes": {}  
> }

(These \_tasks API calls are new to me)

But... The new index now has the same amount of documents like the original one. And the index I used was the one that I earlier established had additional docs - I cleared it out before re-indexing. I did the search for that additional doc but it's not there.

It makes me worried that there might indeed be something changing/removing documents but there is nothing from our side that should be doing it.

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [January 18, 2018, 2:41pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/32 "2018-01-18T14:41:22Z")

</div>

See the second part of my previous comment on how to retrieve the task status after the task has finished running.

I'm pretty confident that it is not the reindex process causing the issue you're seeing, but some rogue process somewhere. We'll find out.

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 2:43pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/33 "2018-01-18T14:43:06Z")

</div>

I used `wait_for_completion=false` as suggested. Unless it was supposed to be set to `true` instead?

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [January 18, 2018, 2:47pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/34 "2018-01-18T14:47:49Z")

</div>

You ran this ?

```
 POST _reindex?wait_for_completion=false

```

If yes, then in the response you get the task ID that you can then use to query its status like this:

```
GET _tasks/<taskID>

```

or like this:

```
GET .tasks/task/<taskID>

```

The second query will give you a document containing the task status even after the task has finished. You're supposed to delete it when you don't need it anymore.

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 2:55pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/35 "2018-01-18T14:55:11Z")

</div>

Aaah, I see. 🙂 All info available now.

> {  
> "completed": true,  
> "task": {  
> "node": "dummy node",  
> "id": 72607957,  
> "type": "transport",  
> "action": "indices:data/write/reindex",  
> "status": {  
> "total": 646865,  
> "updated": 646865,  
> "created": 0,  
> "deleted": 0,  
> "batches": 647,  
> "version\_conflicts": 0,  
> "noops": 0,  
> "retries": {  
> "bulk": 0,  
> "search": 0  
> },  
> "throttled\_millis": 0,  
> "requests\_per\_second": -1,  
> "throttled\_until\_millis": 0  
> },  
> "description": "reindex from [prod-2015.04][order] updated with Script{type=inline, lang='painless', idOrCode='ctx.\_source.remove('account\_is\_subscriber');ctx.\_source.remove('item\_count\_stock\_tracked');', options={}, params={}} to [orders\_prod-2015.04]",  
> "start\_time\_in\_millis": 1516285516650,  
> "running\_time\_in\_nanos": 224687658206,  
> "cancellable": true  
> },  
> "response": {  
> "took": 224687,  
> "timed\_out": false,  
> "total": 646865,  
> "updated": 646865,  
> "created": 0,  
> "deleted": 0,  
> "batches": 647,  
> "version\_conflicts": 0,  
> "noops": 0,  
> "retries": {  
> "bulk": 0,  
> "search": 0  
> },  
> "throttled\_millis": 0,  
> "requests\_per\_second": -1,  
> "throttled\_until\_millis": 0,  
> "failures":   
> }  
> }

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [January 18, 2018, 2:56pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/36 "2018-01-18T14:56:30Z")

</div>

Ok great job! Now, does `646865` correspond to the number of documents you have in the source index `prod-2015.04` (only the `order` type of course)?

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 2:57pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/37 "2018-01-18T14:57:06Z")

</div>

Yes, it does.

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [January 18, 2018, 2:58pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/38 "2018-01-18T14:58:10Z")

</div>

There we go, we've been looking at the wrong place all this time 🙂  
The reindex process doesn't seem to be the culprit, I'm afraid.

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 3:01pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/39 "2018-01-18T15:01:41Z")

</div>

Well, this makes it even greater mystery as other than myself and my manager (who was away while I was re-indexing data) there is no one else who plays with elasticsearch for the project I am in. 😂

---

<div class="post-metadata">

**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\
**Post date:** [January 18, 2018, 3:02pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/40 "2018-01-18T15:02:16Z")

</div>

Thank you for your time. I really appreciate it. 🙂

[Previous page](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015.md?page=1)

[Next page](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015.md?page=3)
