# Doc.deleted keeps increasing while indexing

**URL:** <https://discuss.elastic.co/t/doc-deleted-keeps-increasing-while-indexing/68725>\
**Category:** Elasticsearch\
**Created:** [December 12, 2016, 2:46pm UTC](https://discuss.elastic.co/t/doc-deleted-keeps-increasing-while-indexing/68725 "2016-12-12T14:46:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdsrosa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mdsrosa/32/13808_2.png) [@mdsrosa](https://discuss.elastic.co/u/mdsrosa)\
**Post date:** [December 12, 2016, 2:46pm UTC](https://discuss.elastic.co/t/doc-deleted-keeps-increasing-while-indexing/68725/1 "2016-12-12T14:46:11Z")

</div>

Hi guys,

I'm trying to index 24mi items into ES and I noticed that the number of `doc.deleted` started to increase after reaching 1mi even though I'm not doing any `update`/`delete` calls. I'm using the Bulk API for Python with elasticsearch-py.

My current Elasticsearch version is 2.3 running on AWS.

Here are some calls I did on `/_cat/indices?v`:

```
health status index pri rep docs.count docs.deleted store.size pri.store.size 
yellow open .kibana-4 1 1 2 0 7.8kb 7.8kb 
yellow open events 5 1 1000000 0 92.6mb 92.6mb 
---------------------------
health status index pri rep docs.count docs.deleted store.size pri.store.size 
yellow open .kibana-4 1 1 2 0 7.8kb 7.8kb 
yellow open events 5 1 1057152 0 103.4mb 103.4mb 
---------------------------
health status index pri rep docs.count docs.deleted store.size pri.store.size 
yellow open .kibana-4 1 1 2 0 7.8kb 7.8kb 
yellow open events 5 1 2201952 67600 310.3mb 310.3mb 
--------------------------
health status index pri rep docs.count docs.deleted store.size pri.store.size 
yellow open .kibana-4 1 1 2 0 7.8kb 7.8kb 
yellow open events 5 1 2728081 116471 304.1mb 304.1mb
```

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [December 12, 2016, 8:15pm UTC](https://discuss.elastic.co/t/doc-deleted-keeps-increasing-while-indexing/68725/2 "2016-12-12T20:15:23Z")

</div>

You might be indexing to the same document id. An update is just a delete  
followed by an insert.

---

<div class="post-metadata">

**Author:** ![mdsrosa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mdsrosa/32/13808_2.png) [@mdsrosa](https://discuss.elastic.co/u/mdsrosa)\
**Post date:** [December 13, 2016, 12:01am UTC](https://discuss.elastic.co/t/doc-deleted-keeps-increasing-while-indexing/68725/3 "2016-12-13T00:01:02Z")

</div>

Yeah. The problem was that I was not ordering the items. What I was doing was getting the data from PostgreSQL with pagination but without ordering the items before doing the pagination so each page could have the same items as the previous page and that's why it was getting deleted.

Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2017, 12:01am UTC](https://discuss.elastic.co/t/doc-deleted-keeps-increasing-while-indexing/68725/4 "2017-01-10T00:01:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.