Soft_deletes enabled maybe slow down the index performance

[hackerwin7]

Hi,

Our elasticsearch is on 6.8.0

when we use soft_delete is enabled, we find that the index which indexing requests is mainly updates, so the segments bytes grow up more and more bigger, and docs.deleted count is more and more higher.

because index.soft_deletes.retention_lease.period is default 12h, the deleted docs still live in segments after merges.

this cause to the index most update requests slow down and timeout.

I want to know is there any issues or improvements for this in 7,x or 6.8.x version of Elasticsearch?

[DavidTurner]

Are you using CCR? If so, do you have a follower that has been offline for a long time?

If the answer to either question is "no" then no extra operations are being retained due to soft deletes.

[hackerwin7]

Yes, we use CCR, but the follower index is active.and we shorten the index.soft_deletes.retention_lease.period to resolve this issue.

[hackerwin7]

What is a follower offline mean? the follower index status is green.

[DavidTurner]

Ok, probably best to open a ticket with your support engineer to investigate further. There have been a handful of bugs fixed in the 6.8 series in the area of retention leases, and a good deal of further development in the 7.x series too.

[hackerwin7]

Thanks David,

one more interesting thing is the only one index is huge, but other indices with soft_deletes enabled have no problems.

maybe this unique follower index have leader index history retention lease issue?

be appreciate it if you provide some related issues with this from 6.8.0

[DavidTurner]

Yes possibly this is a bug, and possibly it's fixed in an upgrade -- a lot has changed in the 18 months since 6.8.0 was released. Without digging deeper we can only speculate on what's causing it.