Hello. I have a separate management tool which is handling what ILM would refer to as my "hot" and "warm" phases. I'm looking to invoke an ILM policy to only handle my "cold" phase - which would freeze my indices after a certain number of days. The execution of the freeze seems to fail because the index is read_only.
Is my proposed strategy wholly incompatible with ILM policies? I notice ILM's "warm" phase has an action to toggle on readonly, but there doesn't seem to be any phase/action combination that will toggle off readonly - so that the freeze execution can take place. There must be something under the hood that handles this - because how else would someone relying exclusively on ILM policies get from warm to cold without readonly being toggled off somehow?
I setup my policy for this:
curl -XPUT 'localhost:9200/_ilm/policy/freeze_after_150days?pretty' -H 'Content-Type: application/json' -d'
{
"policy": {
"phases": {
"cold": {
"min_age": "150d",
"actions": {
"freeze": {},
"set_priority": {
"priority": 0
}
}
}
}
}
}'
And then associated that policy with my index (which is already read_only)
curl -XPUT 'localhost:9200/elasticsearch-2019.12.27/_settings' -H 'Content-Type: application/json' -d'{ "index.blocks.read_only": "false" }'
curl -XPUT 'localhost:9200/elasticsearch-2019.12.27/_settings' -H 'Content-Type: application/json' -d'{ "index.lifecycle.name": "freeze_after_150days" }'
curl -XPUT 'localhost:9200/elasticsearch-2019.12.27/_settings' -H 'Content-Type: application/json' -d'{ "index.blocks.read_only": "true" }'
^(This recreated how my indices would be in the wild. Because they would have had the ILM policy associated with them on creation via the index template, and exist in their warm phase with a read_only setting.)
The execution of the policy seems to fail, as evidenced via the _ilm/explain API:
curl -XGET 'http://localhost:9200/elasticsearch-2019.12.27/_ilm/explain'
{
"indices": {
"elasticsearch-2019.12.27": {
"index": "elasticsearch-2019.12.27",
"managed": true,
"policy": "freeze_after_150days",
"lifecycle_date_millis": 1577455107610,
"phase": "cold",
"phase_time_millis": 1599247957876,
"action": "set_priority",
"action_time_millis": 1599247957876,
"step": "ERROR",
"step_time_millis": 1599247958056,
"failed_step": "set_priority",
"step_info": {
"type": "cluster_block_exception",
"reason": "index [elasticsearch-2019.12.27] blocked by: [FORBIDDEN/5/index read-only (api)];",
"stack_trace": "ClusterBlockException[index [elasticsearch-2019.12.27] blocked by: [FORBIDDEN/5/index read-only (api)];]\n\tat org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:222)\n\tat org.elasticsearch.action.admin.indices.settings.put.TransportUpdateSettingsAction.checkBlock(TransportUpdateSettingsAction.java:72)\n\tat org.elasticsearch.action.admin.indices.settings.put.TransportUpdateSettingsAction.checkBlock(TransportUpdateSettingsAction.java:40)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:170)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:161)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:138)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64)\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83)\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394)\n\tat org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:89)\n\tat org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:152)\n\tat org.elasticsearch.xpack.indexlifecycle.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:51)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394)\n\tat org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1214)\n\tat org.elasticsearch.client.support.AbstractClient$IndicesAdmin.updateSettings(AbstractClient.java:1590)\n\tat org.elasticsearch.xpack.core.indexlifecycle.UpdateSettingsStep.performAction(UpdateSettingsStep.java:34)\n\tat org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.maybeRunAsyncAction(IndexLifecycleRunner.java:176)\n\tat org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.lambda$moveToStep$0(IndexLifecycleRunner.java:465)\n\tat org.elasticsearch.xpack.indexlifecycle.MoveToNextStepUpdateTask.clusterStateProcessed(MoveToNextStepUpdateTask.java:83)\n\tat org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.clusterStateProcessed(MasterService.java:520)\n\tat org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$processedDifferentClusterState$1(MasterService.java:407)\n\tat java.base/java.util.ArrayList.forEach(ArrayList.java:1540)\n\tat org.elasticsearch.cluster.service.MasterService$TaskOutputs.processedDifferentClusterState(MasterService.java:407)\n\tat org.elasticsearch.cluster.service.MasterService.onPublicationSuccess(MasterService.java:264)\n\tat org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:257)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:835)\n"
},
"phase_execution": {
"policy": "freeze_after_150days",
"phase_definition": {
"min_age": "150d",
"actions": {
"freeze": {},
"set_priority": {
"priority": 0
}
}
},
"version": 1,
"modified_date_in_millis": 1599247957463
}
}
}
}