Security-exception-action-[indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-error-000001]

I have the elasticsearch,kibana,apm-server in a single ec2 instance.
Able to get apm agent data from other application servers.

When I look into Index Management I can see index errors for all apm indices

ilm.step:ERROR

apm-7.6.0-error-000001
apm-7.6.0-span-000001
apm-7.6.0-profile-000001
apm-7.6.0-metric-000001
apm-7.6.0-transaction-000001

_GET /apm-7.6.0-span-000001/_ilm/explain

      "step_info" : {
        "type" : "security_exception",
        "reason" : "action [indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-span-000001], this action is granted by the index privileges [manage,all]",
        "stack_trace" : """ElasticsearchSecurityException[action [indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-span-000001], this action is granted by the index privileges [manage,all]]

Error shows that I am using kibana user for apm-server which dont have ilm access,but I am using a separate user 'apm-server-kibana' with kibana_system,kibana_admin,apm_system,apm-ilm roles..I have added 'all'access for ilm for apm* indices using apm-ilm role .

In apm-server.yml I am not using user 'kibana' anywhere but using 'apm-server-kibana'

Why this error shows as Kibana user?

How to fix this error?

Stack trace :

ElasticsearchSecurityException[action [indices:admin/settings/update] is unauthorized for user [kibana] on indices [apm-7.6.0-error-000001], 
        this action is granted by the index privileges [manage,all]]
    at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:35)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:656)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.access$300(AuthorizationService.java:101)
    at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:704)
    at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:689)
    at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:659)
    at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
    at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:556)
    at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:336)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
    at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:722)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:599)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:290)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
    at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:722)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
    at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:367)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:286)
    at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:720)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:289)
    at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:720)
    at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$5(RBACEngine.java:328)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
    at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:352)
    at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:325)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:300)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:265)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:229)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
    at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
    at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:127)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
    at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:161)
    at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:278)
    at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:133)
    at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:121)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:231)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:330)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:391)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:402)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:327)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:268)
    at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:161)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:154)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:106)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77)
    at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86)
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:66)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402)
    at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:196)
    at org.elasticsearch.xpack.ilm.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:52)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1286)
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.updateSettings(AbstractClient.java:1672)
    at org.elasticsearch.xpack.core.ilm.UpdateSettingsStep.performAction(UpdateSettingsStep.java:42)
    at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.maybeRunAsyncAction(IndexLifecycleRunner.java:290)
    at org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.clusterStateProcessed(IndexLifecycleRunner.java:246)
    at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.clusterStateProcessed(MasterService.java:523)
    at org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$processedDifferentClusterState$1(MasterService.java:410)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.elasticsearch.cluster.service.MasterService$TaskOutputs.processedDifferentClusterState(MasterService.java:410)
    at org.elasticsearch.cluster.service.MasterService.onPublicationSuccess(MasterService.java:270)
    at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:262)
    at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:239)
    at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:62)
    at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:140)
    at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:139)
    at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:177)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:241)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:204)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)

In Kibana.yml

elasticsearch.username: kibana
-> I have changed this user as different user and tried but still the apm indices are showing as 'kibana' user

These apm rollover policies are created by default when using apm and these policies uses the default user 'kibana' to create it.. So Kibana user dont have access for update.

So as per documentation line if I modify the default apm rollover policy with the logged in user[having access for update ilm],then select the 'retry index' option has solved this error.

Documentation:

If you use Elasticsearch’s security features, ILM performs operations as the user who last updated the policy. ILM only has the roles assigned to the user at the time of the last policy update.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.