Index lifecycle not working because of "FORBIDDEN/12/index read-only / allow delete"

Hello,

I am still struggling with the index lifecycle management on my elastic.co cluster.
in the index management section of kibana, I have some errors:
3 indices have lifecycle errors I focus to one of them.

Effectively, I have set a rollup policy to 20Gb, and I am over (
24.1gb currently). So something is not OK.

When I navigate to the details, I can see this
cluster_block_exception: index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Some discussions and doc says (and I did it)

PUT /filebeat-7.5.1-*/_settings
{
  "index.blocks.read_only_allow_delete": null
}

which I have, I suppose:

filebeat-7.5.1-2020.01.10-000001 settings extract

      "blocks": {
        "metadata": "false",
        "read": "false",
        "read_only_allow_delete": "false",
        "read_only": "false",
        "write": "false"
      },

What's the solution?

best

This appears to have happened because your node reached the "flood stage" in terms of its disk being almost full (see: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/disk-allocator.html )

Assuming that you fixed the disk issue, where is ILM currently? Can you do GET /<index-name>/_ilm/explain?human and send the output so we can see where the index currently is in the ILM policy?

I have cut the JSON so that it fit in the form

{
  "indices" : {
    "filebeat-7.5.1-2020.01.10-000001" : {
      "index" : "filebeat-7.5.1-2020.01.10-000001",
      "managed" : true,
      "policy" : "filebeat-7.5.1",
      "lifecycle_date" : "2020-01-10T10:47:01.095Z",
      "lifecycle_date_millis" : 1578653221095,
      "age" : "68.2d",
      "phase" : "hot",
      "phase_time" : "2020-01-10T10:47:01.266Z",
      "phase_time_millis" : 1578653221266,
      "action" : "rollover",
      "action_time" : "2020-01-10T10:49:36.726Z",
      "action_time_millis" : 1578653376726,
      "step" : "ERROR",
      "step_time" : "2020-01-20T10:59:36.721Z",
      "step_time_millis" : 1579517976721,
      "failed_step" : "check-rollover-ready",
      "step_info" : {
        "type" : "cluster_block_exception",
        "reason" : "index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];",
        "stack_trace" : """ClusterBlockException[index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
	at org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:222)
	at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:109)
	at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:73)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:141)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:132)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:109)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507)
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329)
.....
"""
      },
      "phase_execution" : {
        "policy" : "filebeat-7.5.1",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 3,
        "modified_date" : "2020-01-10T10:47:00.423Z",
        "modified_date_in_millis" : 1578653220423
      }
    }
  }
}

Okay, so the block should automatically release once the disk goes down to a reasonable level (are you still above the flood threshold of 95% full?).

You can retry the ILM operation using:

POST /<index>/_ilm/retry

Give that a shot (once the disk issue is solved) and that should allow ILM to continue running. In the future ILM actions will be automatically retried (in 7.6 the rollover step will be automatically retried)

1 Like

Hey dakrone,

perfect answer, perfect reactivity, perfect timing for me

thank's a lot, problem solved, and hope 7.6 will help me retrying

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.