# Ways to purge deleted documents in 4TB indice **URL:** \ **Category:** Elasticsearch\ **Created:** [December 1, 2018, 2:37am UTC](https://discuss.elastic.co/t/ways-to-purge-deleted-documents-in-4tb-indice/158988 "2018-12-01T02:37:03Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![orivera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orivera/32/37223_2.png) [@orivera](https://discuss.elastic.co/u/orivera)\ **Post date:** [December 1, 2018, 2:37am UTC](https://discuss.elastic.co/t/ways-to-purge-deleted-documents-in-4tb-indice/158988/1 "2018-12-01T02:37:03Z")
Looking for an advice on how to get rid of all deleted documents in 4TB indice on a monthly basis. This indice is not a time series and it has lots deletes and writes all day long. These are the number of docs from [\_stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html) ``` "primaries" : { "docs" : { "count" : 4377351691, "deleted" : 1276015486 }, "total" : { "docs" : { "count" : 8754708022, "deleted" : 2565553924 }, ``` When I look at the [\_segments](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-segments.html) output I can see some segments with 40% data deletes, here is a sample [https://gist.githubusercontent.com/ofrivera/a16d67cfa4e3b59c21db1a2b4e8615c7/raw/754e1d46ab468e448dbe40e3b0a3d6208f1706b0/gistfile1.txt](https://gist.githubusercontent.com/ofrivera/a16d67cfa4e3b59c21db1a2b4e8615c7/raw/754e1d46ab468e448dbe40e3b0a3d6208f1706b0/gistfile1.txt) Some options I'm considering: - Create a copy of the indice (via snapshot/restore), convert to read only, expunge deletes, sync deltas. - [\_forcemerge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) but what options you suggest?, specially to prevent ending up with huge segments. - Or just trying to be more aggressive with [index.merge.scheduler.max\_thread\_count](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-merge.html) any suggestion? Some details about cluster setup: Specs ``` AWS i3.4xlarge 16 nodes (all data nodes) 120gb RAM memory 16cpus ``` Health ```auto { "cluster_name" : "elasticsearch", "status" : "green", "timed_out" : false, "number_of_nodes" : 16, "number_of_data_nodes" : 16, "active_primary_shards" : 629, "active_shards" : 1258, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` OS CentOS Linux release 7.5.1804 (Core) 3.10.0-862.14.4.el7.x86\_64 #1 SMP Wed Nov 30 15:12:11 UTC 2018 x86\_64 x86\_64 x86\_64 GNU/Linux RPM: elasticsearch-6.4.2.rpm Details: elasticsearch-6.4.2-1.noarch JAVA: openjdk version "1.8.0\_191" OpenJDK Runtime Environment (build 1.8.0\_191-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) Thanks! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 1, 2018, 9:51am UTC](https://discuss.elastic.co/t/ways-to-purge-deleted-documents-in-4tb-indice/158988/2 "2018-12-01T09:51:51Z")
> [@orivera](#): > > When I look at the [\_segments](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-segments.html) output I can see some segments with 40% data deletes The total proportion of deletes in your index is ~25% which seems healthy for an index that is under load. > [@orivera](#): > > Looking for an advice on how to get rid of all deleted documents in 4TB indice on a monthly basis. Importantly, why do you want to do this? The answer depends on the goal you're trying to achieve. ---
**Author:** ![orivera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orivera/32/37223_2.png) [@orivera](https://discuss.elastic.co/u/orivera)\ **Post date:** [December 3, 2018, 6:49pm UTC](https://discuss.elastic.co/t/ways-to-purge-deleted-documents-in-4tb-indice/158988/3 "2018-12-03T18:49:55Z")
> [@DavidTurner](#): > > > Looking for an advice on how to get rid of all deleted documents in 4TB indice on a monthly basis. > > Importantly, why do you want to do this? The answer depends on the goal you're trying to achieve. We have noticed, that the cluster performance improves when we keep the % of deleted documents low, specially number of slow queries and queued tasks decreases, we have been using 2 ways in the past, using [\_forcemerge?only\_expunge\_deletes=true](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) and [reindex](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html), but as the indice continues to grow this becomes a more challenging task. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 3, 2018, 7:09pm UTC](https://discuss.elastic.co/t/ways-to-purge-deleted-documents-in-4tb-indice/158988/4 "2018-12-03T19:09:16Z")
`forcemerge` can "upset" the algorithm that selects which segments to merge so it tends to cause more trouble in the long run unless the data set isn't changing. I believe some folks have done some recent work on making it not quite so bad upsetting to the algorithm, but I don't have a handy link and couldn't find it when it looked. Sorry! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 3, 2018, 9:53pm UTC](https://discuss.elastic.co/t/ways-to-purge-deleted-documents-in-4tb-indice/158988/5 "2018-12-03T21:53:08Z")
@nik9000 possibly you are thinking of [https://issues.apache.org/jira/browse/LUCENE-7976](https://issues.apache.org/jira/browse/LUCENE-7976) and/or [https://github.com/elastic/elasticsearch/issues/32323](https://github.com/elastic/elasticsearch/issues/32323)? I'm no expert in this area, but `_forcemerge` today seems like a bad idea because it will result in unnaturally large segments that are correspondingly harder to purge of deletes. It seems that changes are in the works so this may not be true in future. Changing a thread count might help if your system were falling behind with its merges, but since you have a reasonable proportion of deletes it seems plausible that it's exactly where it wants to be. That leaves the reindex idea. ---
**Author:** ![orivera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orivera/32/37223_2.png) [@orivera](https://discuss.elastic.co/u/orivera)\ **Post date:** [December 15, 2018, 5:07pm UTC](https://discuss.elastic.co/t/ways-to-purge-deleted-documents-in-4tb-indice/158988/6 "2018-12-15T17:07:08Z")
Thanks guys we decided to go with reindex option. Also I would recommend the following 2 articles for people facing this issue. > **[Lucene's Handling of Deleted Documents | Elastic](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents)** > > When a document is deleted or updated (= delete + add), Apache Lucene simply marks a bit in a per-segment bitset to record that the document is deleted. All subsequent searches simply skip any deleted... > **[How many shards should I have in my Elasticsearch cluster? | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet... Be aware that some of the options mentioned on the links above like `index.merge.policy.max_merged_segment` and `index.merge.policy.reclaim_deletes_weight` seem to be already deprecated at [Merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-merge.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 12, 2019, 5:07pm UTC](https://discuss.elastic.co/t/ways-to-purge-deleted-documents-in-4tb-indice/158988/7 "2019-01-12T17:07:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.