I am still struggling with the index lifecycle management on my elastic.co cluster.
in the index management section of kibana, I have some errors: 3 indices have lifecycle errors I focus to one of them.
Effectively, I have set a rollup policy to 20Gb, and I am over (
24.1gb currently). So something is not OK.
When I navigate to the details, I can see this cluster_block_exception: index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
Some discussions and doc says (and I did it)
PUT /filebeat-7.5.1-*/_settings
{
"index.blocks.read_only_allow_delete": null
}
Assuming that you fixed the disk issue, where is ILM currently? Can you do GET /<index-name>/_ilm/explain?human and send the output so we can see where the index currently is in the ILM policy?
{
"indices" : {
"filebeat-7.5.1-2020.01.10-000001" : {
"index" : "filebeat-7.5.1-2020.01.10-000001",
"managed" : true,
"policy" : "filebeat-7.5.1",
"lifecycle_date" : "2020-01-10T10:47:01.095Z",
"lifecycle_date_millis" : 1578653221095,
"age" : "68.2d",
"phase" : "hot",
"phase_time" : "2020-01-10T10:47:01.266Z",
"phase_time_millis" : 1578653221266,
"action" : "rollover",
"action_time" : "2020-01-10T10:49:36.726Z",
"action_time_millis" : 1578653376726,
"step" : "ERROR",
"step_time" : "2020-01-20T10:59:36.721Z",
"step_time_millis" : 1579517976721,
"failed_step" : "check-rollover-ready",
"step_info" : {
"type" : "cluster_block_exception",
"reason" : "index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];",
"stack_trace" : """ClusterBlockException[index [filebeat-7.5.1-2020.01.10-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
at org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:222)
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:109)
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:73)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:141)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:132)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:109)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:58)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:172)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)
at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345)
at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263)
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507)
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329)
.....
"""
},
"phase_execution" : {
"policy" : "filebeat-7.5.1",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "30d"
}
}
},
"version" : 3,
"modified_date" : "2020-01-10T10:47:00.423Z",
"modified_date_in_millis" : 1578653220423
}
}
}
}
Okay, so the block should automatically release once the disk goes down to a reasonable level (are you still above the flood threshold of 95% full?).
You can retry the ILM operation using:
POST /<index>/_ilm/retry
Give that a shot (once the disk issue is solved) and that should allow ILM to continue running. In the future ILM actions will be automatically retried (in 7.6 the rollover step will be automatically retried)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.