I have created an Index of 200M docs and will be updated frequently. 50M docs will be updated in a month. In short, the Index is read/write heavy in nature.
As Elastic/Lucene says it will not do an actual update but it will delete/add, that means the deleted docs will resides in Index but will not be searchable.
Lucene occasionally merges segments according to merge policy, which is costlier.
So my question is
- How would be my Index read/write performant in such scenario?
- Is there any alternative to deal with such scenario?
- Can we disable soft delete in Elastic/Lucene and allow only hard delete?