Some question about soft_deletes and Reindex API

Elastic Stack Elasticsearch
1

Two questions

  1. How to use reindex API with zero downtime.
  2. Could the index setting of index.soft_deletes_enabled be non-dynamic and replace reindex API with setting index.soft_deletes_enabled to true for a exist index witch needs soft_deletes enabled.

where are Questions from

  • when i read the document of Elasticsearch, there is a word come up with me that "Soft deletes can only be configured at index creation, if you want to enabled it for a exist index, you must reindex your data into a new index with soft deletes enabled".

  • I have read some paper of Lucene about the feature of soft deletes. And I have do some test with Lucene that it work well when using hard delete and soft delete in the mean time. So I think hard delete is not conflicted with soft delete in Lucene.

  • In the Elasticsearch level, index.soft_deletes_enabled is a IndexScope.Final setting. And I ask one of authors of Elasticsearch, he tells me that the reason why made it final is in order to siplify model.

  • So I do some test with Elasticsearch in v6.8.3, download the souce code and modify setting of index.soft_deletes.enabled as non-dynamic which look like following:

Setting.boolSetting("index.soft_deletes.enabled", false, Property.IndexScope);

  • It work well for me that enabled index.soft_deletes.enabled for a exist closed index. When I reopen the index, it still work well in CURD and so on.

  • As we know, the reindex API is just reindex snapshot of exist index when it triggers. It will result in inconsistent data between old and new index when reindex then updating data in the old index. So here comes with the first question, is there a best practice to use reindex API with zero downtime.

  • if not, why not for non-dynamic setting of index.soft_deletes.enabled to avoid reindex in this situation. Here comes with the second question I mentions forward.

  • Could anyone explain the two questions in principle. Thanks very much.

1 Like
2

Hey,

I think there is a misunderstanding of soft deletes. Can you explain why and how you link soft deletes and using reindex API?

And also how you link reindex API and zero downtime?

Adding some more context about your goal would help a lot to figure out what you are after (independent from any technology or settings), so that we can maybe go from there.

Side note: Since 7.6.0 the soft deletes setting is deprecated, see Index modules | Elasticsearch Guide [8.1] | Elastic

3

  • My data is now running in the v6.8.3 version of Elasticsearch in the product environment.

  • I am doing research of CCR for high availability in product, and found that it should do reindex to have soft_deletes enabled, so that the index can be applied CCR feature.

4

How does reindex come into play of your research as per your question above? CCR and reindex are very different beasts.

5

  • In order to applied CCR feature with a exist index whose setting of soft_deletes is disabled, it should use reindex API to enabled settting of soft_deletes for the target index. see ccr-leader-requirements

  • As i know, the reindex API is just reindex snapshot of exist index when it triggers. So if the data continue change when reindex triggers, the data will disconsistent when reindex done. The only way i think out is to down the target index then using reindex. So i want to ask if I misunderstand the reindex API. Is there a way to use reindex without downtime

  • As i know, if only using reindex API, it can't. So why not take soft_deletes as non-dynamic setting. As result, i change the Scope.Final of soft_enabled and it work well. But i don't know if there is something wrong i not find for what i do.

  • if change scope of soft_deletes as non-dynamic, it could be more convinient to use CCR.

  • You konw, Since 7.6, the soft deletes setting is deprecated, but it can't scale how many people using Elasticsearch whose version is prior to 7.6 and before 7.x, the default of soft_deletes is disabled. I suppose the most popular version is between 6.5 and 7.5

6

Just a short note to mention that 6.X is EOL, please upgrade :slight_smile:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.