# Count of docs after \_reindex'ing higher than before **URL:** \ **Category:** Elasticsearch\ **Created:** [January 18, 2018, 10:19am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015 "2018-01-18T10:19:39Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 10:19am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/1 "2018-01-18T10:19:39Z")
Hello, I've recently had to reindex all docs due multi type mappings being deprecated. There is one observation that I can't explain... After reindexing (via \_reindex API) I found that new index has more documents than the original one. How is that possible? I narrowed down which documents were new and I tried to search for them in the old index but with no luck... ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 10:21am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/2 "2018-01-18T10:21:29Z")
How did you get the count of documents? ---
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 10:24am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/3 "2018-01-18T10:24:16Z")
I used Kibana's Discovery page which used absolute time frame. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 10:25am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/4 "2018-01-18T10:25:07Z")
What do you get when running this in Dev Tools? ``` GET _cat/indices?v ``` ---
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 10:29am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/5 "2018-01-18T10:29:27Z")
Hmmm... I think the comparison of document count using the above won't work because old index has got multi type mapping. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 10:36am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/6 "2018-01-18T10:36:07Z")
Ok, then what about the following? ``` curl -XGET http://xx.xx.xx:9200/source-index/source-type/_count?pretty curl -XGET http://xx.xx.xx:9200/dest-index/_count?pretty ``` Note that the count we'll see doesn't not include nested documents. Do you have nested types in your mapping? ---
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 10:52am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/7 "2018-01-18T10:52:15Z")
No nested types in our mapping. So example result is: Old index: 1573300 New index: 1573367 ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 10:57am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/8 "2018-01-18T10:57:02Z")
Since the difference is pretty low, it would be interesting to proceed by dichotomy using a `date_histogram` aggregation on both indices and see in which buckets the differences are. Can you run this aggregation on both of your indices and see in which month (you might use `year` or `day` as well) the differences appear, then we can further drill down, until we find the culprit. ``` { "size": 0, "aggs": { "dichotomy": { "date_histogram": { "field": "your_date_field", "interval": "month" } } } } ``` ---
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 11:24am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/9 "2018-01-18T11:24:03Z")
Our indices are monthly already. I did a visualisation which split data on \_index term and nearly every index has got some discrepancies. I also used your suggested script and split yearly data (so e.g. "index-2015\*") by monthly interval and nearly every month has got higher count in the new index. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 11:42am UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/10 "2018-01-18T11:42:16Z")
Ok, then let's take one month and drill down, by day, hour, minute... until we find one doc that is in the destination index but not in the source index. ---
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 12:49pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/11 "2018-01-18T12:49:29Z")
Found an example of a doc that is not listed in the source index but is in the target index. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 12:53pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/12 "2018-01-18T12:53:44Z")
Good! And what does it tell us? Do you have any idea? Do you want to share it? ---
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 12:58pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/13 "2018-01-18T12:58:33Z")
I won't be able to share it as it has information about a customer and a specific order that was placed. What I tried to do was to pick some potentially unique fields, e.g. order value, channel id it came from, and do a search globally, hoping that something would be found but with no success. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 12:59pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/14 "2018-01-18T12:59:40Z")
How do you call the reindex API? Do you mind sharing your command? Are you using an alias as the source index ? ---
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 1:08pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/15 "2018-01-18T13:08:45Z")
Sure, that I can do. > POST \_reindex > { > "source": { > "index": source\_index, > "type": "order" > }, > "dest": { > "index": target\_index, > "pipeline": "rename\_order\_fields" > }, > "script": { > "lang": "painless", > "source": "ctx.\_source.remove('account\_is\_subscriber');ctx.\_source.remove('item\_count\_stock\_tracked');" > } > } Where `rename_order_fields` pipeline is used to remove `order_` prefix from some fields that changed over time. E.g. > PUT \_ingest/pipeline/rename\_order\_fields > { > "description" : "rename order fields", > "processors" : [ > { > "rename": { > "field": "order\_value\_USD", > "target\_field": "value\_USD" > } > } > etc ... > } Also, not using aliases. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 1:15pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/16 "2018-01-18T13:15:29Z")
Is it possible that the document has been deleted in the source index in the meantime (or during the reindex)? Do you have a process that deletes (old/rotten) documents based on some condition? ---
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 1:16pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/17 "2018-01-18T13:16:17Z")
Nope, we don't delete any documents. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 1:18pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/18 "2018-01-18T13:18:50Z")
What does the diff between the document in the new index and the old index tell us? ---
**Author:** ![Kornelia\_Watson](https://avatars.discourse-cdn.com/v4/letter/k/a5b964/32.png) [@Kornelia\_Watson](https://discuss.elastic.co/u/Kornelia_Watson)\ **Post date:** [January 18, 2018, 1:22pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/19 "2018-01-18T13:22:26Z")
Sorry, not sure what you mean by diff. That document doesn't seem to exist - or at least can't get it to surface - in the old index, so other than the fact it exists in one index but not in the other there is nothing else I can compare. ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 18, 2018, 1:24pm UTC](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015/20 "2018-01-18T13:24:01Z")
Yes, sorry. Ok, then the ID is not supposed to change between the source and the target index. Any way to find that document by ID in any other index? [Next page](https://discuss.elastic.co/t/count-of-docs-after-reindexing-higher-than-before/116015.md?page=2)