Hello Team
I am using the Elasticsearch 8.4.2.
I have several indices, that is receiving updates (or deletes). On some indices update operation goes with the optimistic-concurrency-control.
After some time, when the percent of deleted document goes a bit higher then 30, the update query performance degrades. The update operations, that usually executes fast enough (99 percentile is under 100ms) suddenly start respond in 1sec or more.
For now I use cron to execute the _forcemerge with only_expunge_deletes flag during the night. It helps to keep responses under 100ms, but I am worry about this
We recommend only force merging a read-only index (meaning the index is no longer receiving writes). When documents are updated or deleted, the old version is not immediately removed, but instead soft-deleted and marked with a "tombstone". These soft-deleted documents are automatically cleaned up during regular segment merges. But force merge can cause very large (> 5GB) segments to be produced, which are not eligible for regular merges. So the number of soft-deleted documents can then grow rapidly, resulting in higher disk usage and worse search performance. If you regularly force merge an index receiving writes, this can also make snapshots more expensive, since the new documents can’t be backed up incrementally.
According to the lucene java doc:
https://lucene.apache.org/core/9_0_0/core/org/apache/lucene/index/TieredMergePolicy.html
findForcedDeletesMerges should never produce segments greater than maxSegmentSize.
and also to this change in elastic:
More optimal forced merges when max_num_segments is greater than 1
if I understand everything correctly, it's safe to call _forcemerge with only_expunge_deletes flag and I shouldn't worry about the huge segments appearance on indices that continue receiving index or update operations?