ElasticSearch Segment merge not happening when deleted documents count is greater than 50%

[Vignesh_Ramesh]

Elasticsearch version: 7.10.0

I have an elasticsearch index with 8 shards in 8 different nodes and with a document count greater than 25 million documents(nested not included). It's an heavy update index. The document size grows over a period of time because of deleted documents. I did a search on this issue and read blogs like one below which tells a segment will automatically be merged when the deleted docs count in that segment is greater than 50%.

Too many Deleted Docs

Hi @jpountz Thanks a lot for your response. Please let me know if my understanding is correct, increasing reclaim_deletes_weight will favor segments with more deleted docs from the documentation. For the existing segments that have already reached max_merged_segment , will they be affected by this setting also? The reason why I am worried about the deleted documents , is that they are taking up lot of space and I have read these awesome blogs http://blog.mikemccandless.com/2011/02/visualizing-…

I did a /_segment s for the index and found segments like the below

"segments": {
                            "_bbx": {
                                "generation": 14685,
                                "num_docs": 27901732,
                                "deleted_docs": 23290932,
                                "size_in_bytes": 5071187083,
                                "memory_in_bytes": 137008,
                                "committed": true,
                                "search": true,
                                "version": "8.7.0",
                                "compound": false,
                                "attributes": {
                                    "Lucene87StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },

Full response of /_segment call can be found here

segments.txt

In many segments like the one above the deleted_docs count is more than 75% of the num_docs but is still not getting merged . We haven't set any max_merged_segment so the default is 5gb. We also haven't changed any MergePolicy and are using the default ones as of Es version 7.10.0.

Is my understanding correct ?

Any thoughts on this would be helpful. Thanks in advance.

[Christian_Dahlqvist]

@Vignesh_Ramesh

"num_docs": 27901732,
"deleted_docs": 23290932,

I believe you have 23,290,932 deleted documents out of a total of 51,192,664 (27,901,732 + 23,290,932) documents which means 45.5% are deleted in that segment.