ElasticSearch Segment merge not happening when deleted documents count is greater than 50%

Elastic Stack Elasticsearch
1

Elasticsearch version: 7.10.0

I have an elasticsearch index with 8 shards in 8 different nodes and with a document count greater than 25 million documents(nested not included). It's an heavy update index. The document size grows over a period of time because of deleted documents. I did a search on this issue and read blogs like one below which tells a segment will automatically be merged when the deleted docs count in that segment is greater than 50%.

I did a /_segment s for the index and found segments like the below

"segments": {
                            "_bbx": {
                                "generation": 14685,
                                "num_docs": 27901732,
                                "deleted_docs": 23290932,
                                "size_in_bytes": 5071187083,
                                "memory_in_bytes": 137008,
                                "committed": true,
                                "search": true,
                                "version": "8.7.0",
                                "compound": false,
                                "attributes": {
                                    "Lucene87StoredFieldsFormat.mode": "BEST_SPEED"
                                }
                            },

Full response of /_segment call can be found here

In many segments like the one above the deleted_docs count is more than 75% of the num_docs but is still not getting merged . We haven't set any max_merged_segment so the default is 5gb. We also haven't changed any MergePolicy and are using the default ones as of Es version 7.10.0.

Is my understanding correct ?

Any thoughts on this would be helpful. Thanks in advance.

2

I believe you have 23,290,932 deleted documents out of a total of 51,192,664 (27,901,732 + 23,290,932) documents which means 45.5% are deleted in that segment.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.