Index lifecycle not working because of "FORBIDDEN/12/index read-only / allow delete"

dao · March 18, 2020, 10:42am

Hello,

I am still struggling with the index lifecycle management on my elastic.co cluster.
in the index management section of kibana, I have some errors:
3 indices have lifecycle errors I focus to one of them.

Effectively, I have set a rollup policy to 20Gb, and I am over (
24.1gb currently). So something is not OK.

When I navigate to the details, I can see this
cluster_block_exception: index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Some discussions and doc says (and I did it)

PUT /filebeat-7.5.1-*/_settings
{
  "index.blocks.read_only_allow_delete": null
}

which I have, I suppose:

filebeat-7.5.1-2020.01.10-000001 settings extract

      "blocks": {
        "metadata": "false",
        "read": "false",
        "read_only_allow_delete": "false",
        "read_only": "false",
        "write": "false"
      },

What's the solution?

best

dakrone · March 18, 2020, 2:29pm

This appears to have happened because your node reached the "flood stage" in terms of its disk being almost full (see: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/disk-allocator.html )

Assuming that you fixed the disk issue, where is ILM currently? Can you do GET /<index-name>/_ilm/explain?human and send the output so we can see where the index currently is in the ILM policy?

dao · March 18, 2020, 3:39pm

I have cut the JSON so that it fit in the form

{
  "indices" : {
    "filebeat-7.5.1-2020.01.10-000001" : {
      "index" : "filebeat-7.5.1-2020.01.10-000001",
      "managed" : true,
      "policy" : "filebeat-7.5.1",
      "lifecycle_date" : "2020-01-10T10:47:01.095Z",
      "lifecycle_date_millis" : 1578653221095,
      "age" : "68.2d",
      "phase" : "hot",
      "phase_time" : "2020-01-10T10:47:01.266Z",
      "phase_time_millis" : 1578653221266,
      "action" : "rollover",
      "action_time" : "2020-01-10T10:49:36.726Z",
      "action_time_millis" : 1578653376726,
      "step" : "ERROR",
      "step_time" : "2020-01-20T10:59:36.721Z",
      "step_time_millis" : 1579517976721,
      "failed_step" : "check-rollover-ready",
      "step_info" : {
        "type" : "cluster_block_exception",
        "reason" : "index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];",
        "stack_trace" : """ClusterBlockException[index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
	at org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:222)
	at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:109)
	at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:73)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:141)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:132)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:109)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
	at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
	at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
	at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
	at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507)
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554)
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
	at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329)
.....
"""
      },
      "phase_execution" : {
        "policy" : "filebeat-7.5.1",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 3,
        "modified_date" : "2020-01-10T10:47:00.423Z",
        "modified_date_in_millis" : 1578653220423
      }
    }
  }
}

dakrone · March 18, 2020, 3:48pm

Okay, so the block should automatically release once the disk goes down to a reasonable level (are you still above the flood threshold of 95% full?).

You can retry the ILM operation using:

POST /<index>/_ilm/retry

Give that a shot (once the disk issue is solved) and that should allow ILM to continue running. In the future ILM actions will be automatically retried (in 7.6 the rollover step will be automatically retried)

dao · March 18, 2020, 4:17pm

Hey dakrone,

perfect answer, perfect reactivity, perfect timing for me

thank's a lot, problem solved, and hope 7.6 will help me retrying

system · April 15, 2020, 4:17pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Issue about index lifecycle management Elasticsearch ilm-index-lifecycle-management	4	660	May 24, 2020
Filebeat Index Lifecycle Management Issue (No Data) Elasticsearch ilm-index-lifecycle-management	12	2727	June 6, 2019
Index lifecycle error Elasticsearch ilm-index-lifecycle-management	6	29540	April 8, 2019
Lifecycle Policies current action blank Elasticsearch ilm-index-lifecycle-management	2	420	May 11, 2020
ES 7.11.2 / Kibana error : index has lifecycle error Elasticsearch ilm-index-lifecycle-management	2	429	June 1, 2021

Index lifecycle not working because of "FORBIDDEN/12/index read-only / allow delete"

Related topics