Is there a known issue with POST /_forcemerge?only_expunge_deletes=true on v6.6?
The upgrade isn't happening anytime soon, since it's out of my control. It's running on prem. Besides, I would imagine, it isn't easy to upgrade a multi-terabyte instance that is backing up a 24/7 operation.
Hi @rgelb I am pretty sure we have work together before as I was at CoreLogic...
So besides making sure that you have a direct connection to Elasticsearch, there's a couple things with force merge that are not obvious sometimes.
force merge happens on the node of the primary shard is and in essence there needs to be enough disk space to do it. If There's not enough disk space and I would say think about Let's just say anywhere from additional 50 to 90% of the primary shard size will need to be available on that node... Otherwise it will not be able to complete... I.e It needs room to build the new segments before I can get rid of the old segments.
Second is with that many segments and depending on what the underlying hardware is, it could take quite some time.... Quite some time.
With the cat segments, you'll be able to see if the new segments are being created... Or not.
You should also be able to use the task API believe to see what is running...or not.
And yes, it'll be a significant task to upgrade but it can be done... at Some point assuming this is a business critical app.. We have experts that can help with that.
@stephenb I never worked at CoreLogic, but I've worked with the company a number of times on various real estate projects. Your name does sound familiar.
The issue with space... There is about 40% of space remaining.
And for each node (last one being the master):
Assuming that is the case, I should still be able to clean up deleted records from an index that only has a few. For example, index newsarticles_web01_dev_637044237457830355 has 7833 indexed documents and 3 deleted ones.