Only when segments merge, deleted documents are actually cleaned up.
But when segment size goes beyond a limit (default: 5 GB), segments are no longer merged. In this case, how do you suggest forced removal of deleted documents in this segment (say 40% docs are deleted) ?
There is an option provided where we can set our own default for "index.merge.policy.max_merged_segment", but that alone doesn't seem to completely help in this situation.
How about exposing a REST action (something along the terms of optimize?only_expunge_deletes) which doesn't depend on segment merge, but just goes ahead and removes all deleted documents. (The new segments are going to be a 1:1 mapping of old segments, but with cleaned up deleted docs).