# Elasticsearch duplicates increase disk size

**URL:** <https://discuss.elastic.co/t/elasticsearch-duplicates-increase-disk-size/194842>\
**Category:** Elasticsearch\
**Created:** [August 12, 2019, 12:26pm UTC](https://discuss.elastic.co/t/elasticsearch-duplicates-increase-disk-size/194842 "2019-08-12T12:26:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\
**Post date:** [August 12, 2019, 12:26pm UTC](https://discuss.elastic.co/t/elasticsearch-duplicates-increase-disk-size/194842/1 "2019-08-12T12:26:53Z")

</div>

Hello.  
I have indexed to the elasticsearch 7 milion + documents using the collisions handling by updating the \_version value.  
What I have expected was that the "\_version" will be incremented every time the document with the same "\_id" was attempted to be loaded and that the storage will be unchanged.  
Is this the expected behaviour from the elasticsearch to store the collided document regardless?  
Or the document is scheduled to be deleted in some time in the future deletion clean up operation?

> #logstash ingestion config  
> output {  
> elasticsearch {  
> hosts =\> ["localhost:9200"]  
> index =\> "field\_increment"  
> document\_id =\> "%{fingerprint}"  
> }  
> }

> #indexed collisions.json content  
> {"document": "value"}  
> {"document": "value"}  
> ...

> {  
> "\_shards": {  
> "total": 2,  
> "successful": 2,  
> "failed": 0  
> },  
> "stats": {  
> "uuid": "MCTuaE2sShKybrl6VHAGIw",  
> "primaries": {  
> "docs": {  
> "count": 1,  
> "deleted": 517707  
> },  
> "store": {  
> "size\_in\_bytes": 354878026  
> },  
> "indexing": {  
> "index\_total": 6600000,  
> "index\_time\_in\_millis": 6711318,  
> "index\_current": 0,  
> "index\_failed": 0,  
> "delete\_total": 0,  
> "delete\_time\_in\_millis": 0,  
> "delete\_current": 0,  
> "noop\_update\_total": 0,  
> "is\_throttled": false,  
> "throttle\_time\_in\_millis": 0  
> },  
> ...

---

<div class="post-metadata">

**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\
**Post date:** [August 13, 2019, 6:00am UTC](https://discuss.elastic.co/t/elasticsearch-duplicates-increase-disk-size/194842/2 "2019-08-13T06:00:14Z")

</div>

The storage will be changed when you update documents. The update operation in Elasticsearch will reindex a new document rather than overriding previous document. The stale documents will be cleaned only when merge happens. The size of storage will fluctuate, I think.

---

<div class="post-metadata">

**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\
**Post date:** [August 13, 2019, 6:16am UTC](https://discuss.elastic.co/t/elasticsearch-duplicates-increase-disk-size/194842/3 "2019-08-13T06:16:41Z")

</div>

That is what I have thought.  
Yesterday I was testing this by loading the same document to have versioning collisions causing the version increment.

Have a look:

 ![collision_1](https://us1.discourse-cdn.com/elastic/original/3X/5/9/5970c9245cad27e8898d78ecf549b2af9f802ef8.png)  
 ![collision_2](https://us1.discourse-cdn.com/elastic/original/3X/2/8/282b8ec73fb82c6937138bfb43fda4508295b55c.png)

The segment merges happened, index memory elasticsearch request cache dropped, but the Disk size continued to grow and remained the same:

 ![collision_3](https://us1.discourse-cdn.com/elastic/original/3X/2/f/2fdd6cfef1ad5037c9c4166d2d0a18d00f98fb0f.png)

Is there a way to reclaim that stored space somehow?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 10, 2019, 6:20am UTC](https://discuss.elastic.co/t/elasticsearch-duplicates-increase-disk-size/194842/4 "2019-09-10T06:20:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.