Hello,
I have an a ES cluster running 1.7.4 with 3 nodes (2 data nodes, and a third master node that doesn't store data).
While the index is constantly updated/added to, it generally should be staying around the same amount of documents. I've noticed that the memory usage lately keeps going up and up and the deleted documents in the index was just constantly climbing.
We had about 450M documents in the index but if I looked in marvel, it shows that about 300M of those are deleted documents. I'd read that normally those deleted documents should be removed when the index merges, but this doesn't seem to be the case.
We have recently re-indexed our data into a new index since I needed to modify the mappings and this new index consumes way less memory, but we are seeing that the deleted documents are just climbing (there are about 400k already).
I've read that its not beneficial to run an optimize on an index that is being updated. Is there some setting that I possibly have that is not allowing documents to be deleted over time?
We are also planning on upgrading soon from 1.7 -> 2.x -> 5.x so not sure if this is just a problem with running an old version or not.