Restoring snapshot in elastic delete some indices once restored

Hello, I have an elastic cluster using ECK, running version 8.4.3, testing and understand snapshot restoration and here is what I did: (the questions are listed a the end, started with steps I did)

  1. Created a snapshot repository using S3
  2. Created an SLM policy
PUT _slm/policy/test-snapshot-policy
{
  "schedule": "0 */30 * * * ?", 
  "name": "<test-snapshot-{now/s{yyyy.MM.dd-hh.mm.ss}}>", 
  "repository": "my_s3_repository", 
  "config": { 
    "indices": ["test-index*"],
    "include_global_state": false
  },
  "retention": { 
    "expire_after": "2d", 
    "min_count": 5, 
    "max_count": 100 
  }
}
  1. Created an ILM (using short min_age for testing purpose)
PUT _ilm/policy/my_policy
{
  "policy": {
    "_meta": {
      "description": "test hot-warm-cold tier",
      "project": {
        "name": "myProject",
        "department": "myDepartment"
      }
    },
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "60s"
          }
        }
      },
      "warm": {
        "min_age": "120s",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "150s",
        "actions": {
          
        }
      },
      "delete": {
        "min_age": "200s",
        "actions": {
          "wait_for_snapshot": {
            "policy": "test-snapshot-policy"
          },
          "delete": {}
        }
      }
    }
  }
}
  1. Created a data stream to test:
PUT /_index_template/my-lifecycle-template
{
  "index_patterns": ["test-index"],
  "data_stream" :{},
  "template": {
    "settings": {
      "index.lifecycle.name": "my_policy",
      "index.number_of_shards": 2
    }
  }
}
  1. and indexed a sample document
POST /test-index/_doc
{
  "message": "test document",
  "@timestamp": "2021-12-14"
}

While waiting the few minutes in order to wait for rollover, I forced the snapshot to be created (to not wait for 30 minutes as the SLM definition), and in the "Snapshot and Restore" UI in Kibana, I can see that my snapshot has snapshotted 9 indices for the data stream test-index.
As stated here, I have to stop the ILM using POST _ilm/stop, restored the snapshot as follow:

POST _snapshot/my_s3_repository/test-snapshot-2022.12.22-xxxxxx-xxxxxx/_restore
{
  "indices": "test-index",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored-$1",
  "index_settings": {
    "index.lifecycle.name": "my_restored_policy",  # to update the ilm as the restored indices have always "my-policy" in their settings once restored
    "index.lifecycle.indexing_complete": "false"
  },
  "ignore_index_settings": ["index.lifecycle.name"] # tried this settings as well to not import meta indices
}

when checking one of the imported index settings, for example the first GET .ds-restored-test-index-2022.12.22-000001/_settings, I see that the lifecycle name is no longer the my_policy:

"settings": {
      "index": {
        "lifecycle": {
          "name": "my_restored_policy",
          "indexing_complete": "false"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_cold,data_warm,data_hot"
            }
          }
        },
        "hidden": "true",
        "number_of_shards": "2",
        "provided_name": ".ds-test-index-2022.12.23-000001",
        "creation_date": "1671805627963",
        "number_of_replicas": "1",
        "uuid": "q6ZqEElsRL2fxh6qy05apg",
        "version": {
          "created": "8040399"
        }
      }
    }

and my_restored_policy:

"policy": {
      "phases": {
        "warm": {
          "min_age": "120s",
          "actions": {
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "cold": {
          "min_age": "150s",
          "actions": {}
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "60s"
            }
          }
        }
      },
      "_meta": {
        "description": "test hot-warm-cold tier",
        "project": {
          "name": "myProject",
          "department": "myDepartment"
        }
      }
    },

and when checking GET restored-test-index/_ilm/explain, the first indices from .ds-test-index-xxxx-000001 to .ds-test-index-xxxx-000006 still have the following message:

#...
 "step_info": {
        "message": "waiting for policy 'test-snapshot-policy' to be executed since Thu Dec 22 10:07:35 GMT 2022"
      },
#...

but once I start back the ILM, first indices, from .ds-test-index-xxxx-000001 to .ds-test-index-xxxx-000006 get deleted immediatly, why? :confused: as the ILM changed. is it because they have in "_tier_preference": "data_cold,data_warm,data_hot" and data been in cold phase get deleted in the my_policy rule, but in my_restored_policy indices don't have the rule to be deleted! Can't understand why this happen even when changing the ILM name when resoring snapshot, and stopping the ILM and start it back not really suitable to automate such a task. Want to understand what's happening behind the scenes at a low level please :grinning_face_with_smiling_eyes:

EDIT1:

When updating the ILM policy my_policy with: (without wait_for_snapshot action in delete phase)

PUT _ilm/policy/my_policy
{
  "policy": {
    "_meta": {
      "description": "test hot-warm-cold tier",
      "project": {
        "name": "myProject",
        "department": "myDepartment"
      }
    },
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "60s"
          }
        }
      },
      "warm": {
        "min_age": "120s",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "150s",
        "actions": {
          
        }
      },
      "delete": {
        "min_age": "200s",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

and stop the ilm, restoring indices from snapshot and starting back the ilm, restored indices didn't get deleted, so is it a bug? because from what I understood using the first test the restored indices inherit in their ILM explain metadata that this index need to be deleted even the ILM has changed when restoring, but still apply the delete phase once starting back the ILM.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.