"step_info" : {
"type" : "security_exception",
"reason" : "action [indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-span-000001], this action is granted by the index privileges [manage,all]",
"stack_trace" : """ElasticsearchSecurityException[action [indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-span-000001], this action is granted by the index privileges [manage,all]]
Error shows that I am using kibana user for apm-server which dont have ilm access,but I am using a separate user 'apm-server-kibana' with kibana_system,kibana_admin,apm_system,apm-ilm roles..I have added 'all'access for ilm for apm* indices using apm-ilm role .
In apm-server.yml I am not using user 'kibana' anywhere but using 'apm-server-kibana'
ElasticsearchSecurityException[action [indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-error-000001],
this action is granted by the index privileges [manage,all]]
at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:35)
at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:656)
at org.elasticsearch.xpack.security.authz.AuthorizationService.access$300(AuthorizationService.java:101)
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:704)
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:689)
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:659)
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:556)
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:336)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:722)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:599)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:290)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:722)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:367)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:286)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:720)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:289)
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:720)
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$5(RBACEngine.java:328)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:352)
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:325)
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:300)
at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:265)
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:229)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:127)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:161)
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:278)
at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:133)
at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:121)
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:231)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:330)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:391)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:402)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:327)
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:268)
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:161)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:154)
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:106)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77)
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:66)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402)
at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:196)
at org.elasticsearch.xpack.ilm.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:52)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1286)
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.updateSettings(AbstractClient.java:1672)
at org.elasticsearch.xpack.core.ilm.UpdateSettingsStep.performAction(UpdateSettingsStep.java:42)
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.maybeRunAsyncAction(IndexLifecycleRunner.java:290)
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.clusterStateProcessed(IndexLifecycleRunner.java:246)
at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.clusterStateProcessed(MasterService.java:523)
at org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$processedDifferentClusterState$1(MasterService.java:410)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.elasticsearch.cluster.service.MasterService$TaskOutputs.processedDifferentClusterState(MasterService.java:410)
at org.elasticsearch.cluster.service.MasterService.onPublicationSuccess(MasterService.java:270)
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:262)
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:239)
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:62)
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:140)
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:139)
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:177)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:241)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:204)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
In Kibana.yml
elasticsearch.username: kibana
-> I have changed this user as different user and tried but still the apm indices are showing as 'kibana' user
These apm rollover policies are created by default when using apm and these policies uses the default user 'kibana' to create it.. So Kibana user dont have access for update.
So as per documentation line if I modify the default apm rollover policy with the logged in user[having access for update ilm],then select the 'retry index' option has solved this error.
Documentation:
If you use Elasticsearch’s security features, ILM performs operations as the user who last updated the policy. ILM only has the roles assigned to the user at the time of the last policy update.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.