Background merging does not delete documents?

Elastic Stack Elasticsearch
1

Hi,
We have an index of about 1010^6 documents. The index is dynamic and there are removals as well. In the info we see that there is around 110^6 deleted documents. My understanding was that these documents would be completely removed during bacground merges, but the number seems to be constant for weeks now.

I also understand that es does recommend against manually running optimize to clean up these documents. Any idea what may cause documents not to be deleted?

Thanks,
Ilija

2

This probably just means your index is in steady state?

I.e., merging does reclaim the deleted, lowering the count, but the new deletions arrive and increase the count.

This post goes into some detail about deletions: https://www.elastic.co/blog/lucenes-handling-of-deleted-documents

Mike McCandless