Why data is inserting in index in delete phase, why not new index ... are we missing any configuration?

Elastic Stack Elasticsearch
,
1

I am facing an issue with Elasticsearch where, even after the rollover phase is successfully completed and a new index is created, data continues to be inserted into the old rollover index instead of the newly created index even though the old rollover index ilm policy transitioned to delete face. The rollover process seems to be working as expected, but the new index is not being recognized for indexing operations.

Environment:

Elasticsearch Version: v 7.17

ilm policy:

PUT _ilm/policy/demo-rollover_policy
{
  "policy": {
    "phases": {
      "hot": {                                
        "actions": {
          "rollover": {
            "max_size": "50kb" 
          }
        }
      },
      "delete": {
        "min_age": "5m",                     
        "actions": {
          "delete": {}                      
        }
      }
    }
  }
}

Index_template_preview:

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "demo-rollover_policy",
          "rollover_alias": "demo"
        }
      }
    },
    "aliases": {},
    "mappings": {}
  }
}

Reproduction Steps:

Create an index with the specified ILM policy and index template.
Monitor the rollover process and ensure that the new index is created.
Attempt to insert data into the rollover index.

Expected Behavior:
Data should be inserted into the newly created index after the rollover when posted data into the old rollover index

Actual Behavior:
Data is still being inserted into the old rollover index, however the the rollover index is transitioned to delete phase.

What I have read in the documentation is that on a successful rollover, ES will automatically create the new index according to the template, switch the write index to the new one and change the phase of the old index based on our lifecycle policy.

My understanding to this process is after the successful rollover, when we post data to the old index that is rolled over, it will be inserted into the new index that is created. Is there any configuration that i am missing and causing this behavior.

Note:
This issue is impacting our data indexing workflow, and we would appreciate any guidance or assistance in resolving this behavior.

Thank you for your attention to this matter.

2

Please provide all the steps you have taken to create the indices and roll them over. That way we can spot any issues and possibly also recreaste the issue.

3
  1. Created a rollover policy using the above request body in demo-rollover-policy.
  2. Then, I created an index template using the above request body in index_template_preview.
  3. Afterward, I created an index named "demo-0000," assigned an alias "demo" to it, and set "is_write_index" to true.

After implementing all the above steps, the rollover policy was successfully assigned to the index named "demo-00001." It successfully rollovers and creates a new index named "demo-00002" with the same alias "demo." After the creation of the new index, the previous index "demo-00001" entered the delete phase. However, when attempting to insert data with the index name "demo-00001," the new data is still being posted to the old index "demo-00001," which should not be the case. The rollover process should have shifted the write_index to "demo-00002."

I don't have any snapshots available right now; otherwise, I would have provided them earlier. I hope this clarifies the scenario through my explanation.

4

You should be posting data to the alias demo not the concrete index demo-00001 That is the point of the write alias, clients write to be the write alias which points to the correct/ new concrete index

2 Likes
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.