Hi,
We've a fairly big elastic cluster (ES version is 2.4.1 and lucene is 5.5.2) having 30 data nodes. We've seen that our biggest index (3.2 TB primary with replication factor of 2 i.e. total ~10 TB. Number of primary shards 18) is having around 55% deleted docs which might happen in certain circumstances. But while analysing the segments for that index we've found that many segments having default max size (5 gb) are having very high number of deleted docs some more than "live" docs, which ideally shouldn't happen as lucene should merge these segments having <50% "live" docs.
We tried forceMerge without any params but nothing much happened. My query is should we try to use expungeDeletes here as trying to merge into bigger segment isn't recommended according to this article also this is a very active index having high number of updates on a high frequency, moreover as an yearly activity we're using delete_by_query for purging old data
One issue why we're skeptical to use "only_expunge_deletes" is because of low disk on cluster currently as few nodes have less than 20% free disk, this also being the reason why we need to reclaim space taken by deleted docs.
Few screenshots
Any suggestions what options do we've?