Security_exception: action [indices:admin/rollover] is unauthorized for user

[hendry.lim (Hendry)]

Elasticsearch: 7.9.0
Kibana: 7.9.0

I am hitting the issue described in #47500.
I believe the same issue has been raised a few times. However, adding manage privilege on the index level for the role does not seem to work. I am still getting the same error.

Screenshots:

Expand

Error

{
    "type": "server",
    "timestamp": "2020-08-19T12:08:46,690Z",
    "level": "WARN",
    "component": "o.e.i.s.IndexShard",
    "cluster.name": "docker-cluster",
    "node.name": "es01",
    "message": " [apm-7.9.0-transaction-000001][0] onPreFetchPhase listener [org.elasticsearch.xpack.security.authz.SecuritySearchOperationListener@ccd1c8b] failed",
    "cluster.uuid": "D8swAaZPTDeLax90eETYwg",
    "node.id": "6a3KukJsTrqiibIoFlOQ8A",
    "stacktrace": ["org.elasticsearch.ElasticsearchSecurityException: [[FAOfBnQBQcojp3HWI0Vd][23968]] expected scroll indices access control [IndicesAccessControl{granted=true, indexPermissions={apm-7.9.0-error=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-span=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-metric=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-onboarding-2020.08.18=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-transaction-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-metric-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-onboarding=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-profile-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-error-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-span-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-profile=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-transaction=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}}}] but found [IndicesAccessControl{granted=true, indexPermissions={apm-7.9.0-error=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-span=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-metric=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-onboarding-2020.08.18=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-transaction-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-metric-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-onboarding=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-profile-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-error-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-span-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-profile=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, apm-7.9.0-transaction=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}}}] in thread context",
        "at org.elasticsearch.xpack.security.authz.SecuritySearchOperationListener.ensureIndicesAccessControlForScrollThreadContext(SecuritySearchOperationListener.java:111) ~[?:?]",
        "at org.elasticsearch.xpack.security.authz.SecuritySearchOperationListener.onPreFetchPhase(SecuritySearchOperationListener.java:95) ~[?:?]",
        "at org.elasticsearch.index.shard.SearchOperationListener$CompositeListener.onPreFetchPhase(SearchOperationListener.java:166) [elasticsearch-7.9.0.jar:7.9.0]",
        "at org.elasticsearch.search.SearchService$SearchOperationListenerExecutor.<init>(SearchService.java:1272) [elasticsearch-7.9.0.jar:7.9.0]",
        "at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$4(SearchService.java:584) [elasticsearch-7.9.0.jar:7.9.0]",
        "at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-7.9.0.jar:7.9.0]",
        "at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-7.9.0.jar:7.9.0]",
        "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.0.jar:7.9.0]",
        "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.9.0.jar:7.9.0]",
        "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) [elasticsearch-7.9.0.jar:7.9.0]",
        "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.0.jar:7.9.0]",
        "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
        "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
        "at java.lang.Thread.run(Thread.java:832) [?:?]"]
}

[hendry.lim (Hendry)]

Added all privilege to the index level privilege, but I am still seeing the same errors.
One interesting thing is that the error only appears for auditbeat, heartbeat, and winlogbeat indices.

I also noticed that sometimes the error resolved by itself.

[MarcinMiszk]

Hi Hedry

Could you tell if you have found a way to solve this problem ?
Our logs are full of warnings/errors like the one mentioned by you.

Kind Regards
Marcin

[hendry.lim (Hendry)]

@MarcinMiszk unfortunately no, it's still happening randomly in my local environment.
I have another ECE environment, which does not seem to be affected by this issue.
I have opened a Github issue #61453.

[hendry.lim (Hendry)]

Error does not appear in 7.9.1. Should have been fixed in #61446.