Index lifecycle not working because of "FORBIDDEN/12/index read-only / allow delete"

Hello,

I am still struggling with the index lifecycle management on my elastic.co cluster.
in the index management section of kibana, I have some errors:
3 indices have lifecycle errors I focus to one of them.

Effectively, I have set a rollup policy to 20Gb, and I am over (
24.1gb currently). So something is not OK.

When I navigate to the details, I can see this
cluster_block_exception: index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Some discussions and doc says (and I did it)

PUT /filebeat-7.5.1-*/_settings
{
  "index.blocks.read_only_allow_delete": null
}

which I have, I suppose:

filebeat-7.5.1-2020.01.10-000001 settings extract

      "blocks": {
        "metadata": "false",
        "read": "false",
        "read_only_allow_delete": "false",
        "read_only": "false",
        "write": "false"
      },

What's the solution?

best

This appears to have happened because your node reached the "flood stage" in terms of its disk being almost full (see: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/disk-allocator.html )

Assuming that you fixed the disk issue, where is ILM currently? Can you do GET /<index-name>/_ilm/explain?human and send the output so we can see where the index currently is in the ILM policy?

I have cut the JSON so that it fit in the form

{
  "indices" : {
    "filebeat-7.5.1-2020.01.10-000001" : {
      "index" : "filebeat-7.5.1-2020.01.10-000001",
      "managed" : true,
      "policy" : "filebeat-7.5.1",
      "lifecycle_date" : "2020-01-10T10:47:01.095Z",
      "lifecycle_date_millis" : 1578653221095,
      "age" : "68.2d",
      "phase" : "hot",
      "phase_time" : "2020-01-10T10:47:01.266Z",
      "phase_time_millis" : 1578653221266,
      "action" : "rollover",
      "action_time" : "2020-01-10T10:49:36.726Z",
      "action_time_millis" : 1578653376726,
      "step" : "ERROR",
      "step_time" : "2020-01-20T10:59:36.721Z",
      "step_time_millis" : 1579517976721,
      "failed_step" : "check-rollover-ready",
      "step_info" : {
        "type" : "cluster_block_exception",
        "reason" : "index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];",
        "stack_trace" : """ClusterBlockException[index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
	at org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:222)
	at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:109)
	at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:73)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:141)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:132)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:109)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507)
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329)
.....
"""
      },
      "phase_execution" : {
        "policy" : "filebeat-7.5.1",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 3,
        "modified_date" : "2020-01-10T10:47:00.423Z",
        "modified_date_in_millis" : 1578653220423
      }
    }
  }
}

Okay, so the block should automatically release once the disk goes down to a reasonable level (are you still above the flood threshold of 95% full?).

You can retry the ILM operation using:

POST /<index>/_ilm/retry

Give that a shot (once the disk issue is solved) and that should allow ILM to continue running. In the future ILM actions will be automatically retried (in 7.6 the rollover step will be automatically retried)

1 Like

Hey dakrone,

perfect answer, perfect reactivity, perfect timing for me

thank's a lot, problem solved, and hope 7.6 will help me retrying