# Purge the deleted documents on disk **URL:** \ **Category:** Elasticsearch\ **Created:** [September 13, 2014, 12:02am UTC](https://discuss.elastic.co/t/purge-the-deleted-documents-on-disk/19768 "2014-09-13T00:02:16Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Wei](https://avatars.discourse-cdn.com/v4/letter/w/90ced4/32.png) [@Wei](https://discuss.elastic.co/u/Wei)\ **Post date:** [September 13, 2014, 12:02am UTC](https://discuss.elastic.co/t/purge-the-deleted-documents-on-disk/19768/1 "2014-09-13T00:02:16Z")
Hi all, If there's any api to clear all the deleted documents on disk? I read that Deleting a document doesn’t immediately remove the document from disk — it just marks it as deleted. Elasticsearch will clean up deleted documents in the background as you continue to index more data. By request the index stats, it shows "primaries": { - "docs": { - "count": 3268352, - "deleted": 71249 } ....} I run refresh, or optimize on my index, But it only cleaned a small number of deleted documents from disk. Is there any way I can clean all the deleted docs? to reduce this number to 0? Thanks Wei -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vineeth\_mohan\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vineeth_mohan_2/32/747_2.png) [@vineeth\_mohan\_2](https://discuss.elastic.co/u/vineeth_mohan_2)\ **Post date:** [September 13, 2014, 3:15am UTC](https://discuss.elastic.co/t/purge-the-deleted-documents-on-disk/19768/2 "2014-09-13T03:15:59Z")
Hello Wei , You can use the in optimize API - max\_num\_segments as 1 or only\_expunge\_deletes as true . OPTIMIZE - > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Thanks Vineeth On Sat, Sep 13, 2014 at 5:32 AM, Wei [wshen@groupon.com](mailto:wshen@groupon.com) wrote: > Hi all, > > If there's any api to clear all the deleted documents on disk? > I read that > Deleting a document doesn’t immediately remove the document from disk — it > just marks it as deleted. > Elasticsearch will clean up deleted documents in the background as you > continue to index more data. > > By request the index stats, it shows > "primaries": { > > - "docs": { > - "count": 3268352, > - "deleted": 71249 > } ....} > > I run refresh, or optimize on my index, But it only cleaned a small number > of deleted documents from disk. Is there any way I can clean all the > deleted docs? to reduce this number to 0? > > Thanks > Wei > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kqh9JRyE77K1%3DgT3U0F7L1S9ERTCCPpz0obKH4Eqa5vw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kqh9JRyE77K1%3DgT3U0F7L1S9ERTCCPpz0obKH4Eqa5vw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Wei](https://avatars.discourse-cdn.com/v4/letter/w/90ced4/32.png) [@Wei](https://discuss.elastic.co/u/Wei)\ **Post date:** [September 14, 2014, 11:19pm UTC](https://discuss.elastic.co/t/purge-the-deleted-documents-on-disk/19768/3 "2014-09-14T23:19:40Z")
Hi Vineeth, Thanks a lot for your response. However I've tried setting max\_num\_segments to 1 and only\_expunge\_deletes to true, using the command curl -XPOST '[http://localhost:9200/\_optimize?only\_expunge\_deletes=true&max\_num\_segments=1](http://localhost:9200/_optimize?only_expunge_deletes=true&max_num_segments=1)' This only cleared some of the deleted documents. e.g, the number of deleted documents reduced from 8k to 7k. But cannot merge all of them. I'm running es v0.20.5. Is there a way I can force merge all deleted documents? p.s. I'm upgrading the ES from 0.20.5 to 1.2.1, shards are damaged after migration, complaining on the write handler to the .del files. I have a feeling this is because of the un-merged deleted documents. Thanks Wei On Friday, September 12, 2014 8:16:25 PM UTC-7, vineeth mohan wrote: > Hello Wei , > > You can use the in optimize API - max\_num\_segments as 1 or > only\_expunge\_deletes as true . > OPTIMIZE - > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html#indices-optimize) > > Thanks > Vineeth > > On Sat, Sep 13, 2014 at 5:32 AM, Wei \<[ws...@groupon.com](mailto:ws...@groupon.com) \\> > wrote: > > > Hi all, > > > > If there's any api to clear all the deleted documents on disk? > > I read that > > Deleting a document doesn’t immediately remove the document from disk — > > it just marks it as deleted. > > Elasticsearch will clean up deleted documents in the background as you > > continue to index more data. > > > > By request the index stats, it shows > > "primaries": { > > > > - "docs": { > > - "count": 3268352, > > - "deleted": 71249 > > } ....} > > > > I run refresh, or optimize on my index, But it only cleaned a small > > number of deleted documents from disk. Is there any way I can clean all the > > deleted docs? to reduce this number to 0? > > > > Thanks > > Wei > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3d566401-c82d-4ec8-bf91-6f4c0a3f2e96%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3d566401-c82d-4ec8-bf91-6f4c0a3f2e96%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\ **Post date:** [September 15, 2014, 9:31am UTC](https://discuss.elastic.co/t/purge-the-deleted-documents-on-disk/19768/4 "2014-09-15T09:31:49Z")
By default Lucene/ES will only merge away the segment if it has "enough" deletes, where "enough" defaults to 10% of the segment. The setting is index.merge.policy.expunge\_deletes\_allowed ... so you can change that if you want to. However I would strongly advise not worrying about this: merging is a very costly operation, and Lucene naturally merges segments over time, favoring segments that have more deletions. Furthermore, deletions are not that costly because deleted docs are filtered out at a very low level in Lucene, before expensive query matching/scoring computations even see them. Only if your index will never change again is it worth forcing merges ... Mike McCandless [http://blog.mikemccandless.com](http://blog.mikemccandless.com) On Sun, Sep 14, 2014 at 7:19 PM, Wei [wshen@groupon.com](mailto:wshen@groupon.com) wrote: > Hi Vineeth, > > Thanks a lot for your response. > > However I've tried setting max\_num\_segments to 1 and only\_expunge\_deletes > to true, using the command > > curl -XPOST ' > [http://localhost:9200/\_optimize?only\_expunge\_deletes=true&max\_num\_segments=1](http://localhost:9200/_optimize?only_expunge_deletes=true&max_num_segments=1) > ' > > This only cleared some of the deleted documents. e.g, the number of > deleted documents reduced from 8k to 7k. But cannot merge all of them. I'm > running es v0.20.5. Is there a way I can force merge all deleted documents? > > p.s. I'm upgrading the ES from 0.20.5 to 1.2.1, shards are damaged after > migration, complaining on the write handler to the .del files. I have a > feeling this is because of the un-merged deleted documents. > Thanks > Wei > > On Friday, September 12, 2014 8:16:25 PM UTC-7, vineeth mohan wrote: > > > Hello Wei , > > > > You can use the in optimize API - max\_num\_segments as 1 or > > only\_expunge\_deletes as true . > > OPTIMIZE - [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > reference/current/indices-optimize.html#indices-optimize > > > > Thanks > > Vineeth > > > > On Sat, Sep 13, 2014 at 5:32 AM, Wei [ws...@groupon.com](mailto:ws...@groupon.com) wrote: > > > > > Hi all, > > > > > > If there's any api to clear all the deleted documents on disk? > > > I read that > > > Deleting a document doesn’t immediately remove the document from disk — > > > it just marks it as deleted. > > > Elasticsearch will clean up deleted documents in the background as you > > > continue to index more data. > > > > > > By request the index stats, it shows > > > "primaries": { > > > > > > - "docs": { > > > - "count": 3268352, > > > - "deleted": 71249 > > > } ....} > > > > > > I run refresh, or optimize on my index, But it only cleaned a small > > > number of deleted documents from disk. Is there any way I can clean all the > > > deleted docs? to reduce this number to 0? > > > > > > Thanks > > > Wei > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/3d566401-c82d-4ec8-bf91-6f4c0a3f2e96%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3d566401-c82d-4ec8-bf91-6f4c0a3f2e96%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/3d566401-c82d-4ec8-bf91-6f4c0a3f2e96%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3d566401-c82d-4ec8-bf91-6f4c0a3f2e96%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAD7smRcP0kyz0uDS\_GGiyr1R%2BjVeaLxeQxfwPypQFpKLyZ2K0A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAD7smRcP0kyz0uDS_GGiyr1R%2BjVeaLxeQxfwPypQFpKLyZ2K0A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:02am UTC](https://discuss.elastic.co/t/purge-the-deleted-documents-on-disk/19768/5 "2017-07-06T01:02:23Z")