ELK-log full with errors

Elastic Stack Elasticsearch
1

Team - The log of ElasticSearch is full with errors (see below).

Most likely, because of this the system is not working as expected.
Any idea what is happening and how to fix this?

It is version 7.9 for Elastic and Kibana.
Beats can be various versions - 7.6 to 7.9.

Thank you - Will

[2020-08-31T22:31:16,679][WARN ][o.e.i.s.IndexShard       ] [elk.itv.lan] [auditbeat-7.9.0-2020.08.28-000001][0] onPreFetchPhase listener [org.elasticsearch.xpack.security.authz.SecuritySearchOperationListener@36669906] failed
org.elasticsearch.ElasticsearchSecurityException: [[Bgw3RnQBQYQGFTWYgG3G][1408887]] expected scroll indices access control [IndicesAccessControl{granted=true, indexPermissions={auditbeat-7.6.2-2020.08.05-000011=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.0-2020.08.04-000008=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.6.2-2020.08.20-000012=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.8.0-2020.08.18-000005=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.1-2020.08.02-000005=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.0-2020.08.19-000009=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.6.2=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.1=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.8.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.1-2020.08.17-000006=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.9.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.8.0-2020.08.03-000004=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.9.0-2020.08.28-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}}}] but found [IndicesAccessControl{granted=true, indexPermissions={auditbeat-7.6.2-2020.08.05-000011=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.0-2020.08.04-000008=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.6.2-2020.08.20-000012=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.8.0-2020.08.18-000005=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.1-2020.08.02-000005=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.0-2020.08.19-000009=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.6.2=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.1=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.8.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.7.1-2020.08.17-000006=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.9.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.8.0-2020.08.03-000004=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, auditbeat-7.9.0-2020.08.28-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}}}] in thread context
	at org.elasticsearch.xpack.security.authz.SecuritySearchOperationListener.ensureIndicesAccessControlForScrollThreadContext(SecuritySearchOperationListener.java:111) ~[?:?]
	at org.elasticsearch.xpack.security.authz.SecuritySearchOperationListener.onPreFetchPhase(SecuritySearchOperationListener.java:95) ~[?:?]
	at org.elasticsearch.index.shard.SearchOperationListener$CompositeListener.onPreFetchPhase(SearchOperationListener.java:166) [elasticsearch-7.9.0.jar:7.9.0]
	at org.elasticsearch.search.SearchService$SearchOperationListenerExecutor.<init>(SearchService.java:1272) [elasticsearch-7.9.0.jar:7.9.0]
	at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$4(SearchService.java:584) [elasticsearch-7.9.0.jar:7.9.0]
	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-7.9.0.jar:7.9.0]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-7.9.0.jar:7.9.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.0.jar:7.9.0]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.9.0.jar:7.9.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) [elasticsearch-7.9.0.jar:7.9.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.0.jar:7.9.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
2

I am having the same issue. Elastic and Kibana are also on 7.9. Most Beats should be 7.9, but there may be a few stragglers that have not yet updated and may be on 7.8.0.

Your issue keeps calling out Auditbeat indices, but mine is calling out Heartbeat indices.

[2020-09-02T07:15:06,083][WARN ][o.e.i.s.IndexShard       ] [ElasticData01] [heartbeat-7.9.0-2020.08.20-000001][0] onPreFetchPhase listener [org.elasticsearch.xpack.security.authz.SecuritySearchOperationListener@256cfac9] failed
org.elasticsearch.ElasticsearchSecurityException: [[Y9orT3QBDv1cS7Tg0uBN][2006027]] expected scroll indices access control [IndicesAccessControl{granted=true, indexPermissions={heartbeat-7.9.0-2020.08.20-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.7.0-2020.06.21-000002=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.8.0-2020.07.29-000002=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.8.0-2020.08.28-000003=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.8.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.8.0-2020.06.29-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.9.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.7.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.7.0-2020.05.22-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}}}] but found [IndicesAccessControl{granted=true, indexPermissions={heartbeat-7.9.0-2020.08.20-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.7.0-2020.06.21-000002=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.8.0-2020.07.29-000002=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.8.0-2020.08.28-000003=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.8.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.8.0-2020.06.29-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.9.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.7.0=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}, heartbeat-7.7.0-2020.05.22-000001=IndexAccessControl{granted=true, fieldPermissions=org.elasticsearch.xpack.core.security.authz.permission.FieldPermissions@1120a, documentPermissions=DocumentPermissions [queries=null, scopedByQueries=null]}}}] in thread context
        at org.elasticsearch.xpack.security.authz.SecuritySearchOperationListener.ensureIndicesAccessControlForScrollThreadContext(SecuritySearchOperationListener.java:111) ~[?:?]
        at org.elasticsearch.xpack.security.authz.SecuritySearchOperationListener.onPreFetchPhase(SecuritySearchOperationListener.java:95) ~[?:?]
        at org.elasticsearch.index.shard.SearchOperationListener$CompositeListener.onPreFetchPhase(SearchOperationListener.java:166) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.search.SearchService$SearchOperationListenerExecutor.<init>(SearchService.java:1272) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$4(SearchService.java:584) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:710) [elasticsearch-7.9.0.jar:7.9.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.0.jar:7.9.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
3

This is a spurious log warn message, it is not indicative of any problems. This is fixed in 7.9.1 .
Apologies for the inconvenience.

4

Thank you, @Albert_Zaharovits. Glad to hear it is not an actual issue. Do you know when 7.9.1 will reach GA?

5

Perhaps true - but in that case it prevents troubleshooting a real issue: no Elastic and no Kibana => yes - the processes are running. But there is no login for Kibana. And the Beats-services can not connect to ElasticSearch - connection refused.

All this started with version 7.9 => until the 7.8-, most of the things are working as expected.

Any suggestions?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.