Randomly encountered auth errors in Kibana severity increased

[FALEN (Ali Can Saykal)]

Hi Community,

I'm randomly encountering auth errors as shared below, severity of these errors are increased and now i cannot stay logged in for 1-2 minutes before encountering one

security_exception Root causes: security_exception: missing authentication credentials for REST request [/_security/user/_has_privileges]

Version: 8.12.2
Build: 70281
Error
    at fetch_Fetch.fetchResponse (https://***.com/70281/bundles/core/core.entry.js:1:277774)
    at async https://***.com/70281/bundles/core/core.entry.js:1:275707
    at async https://***.com/70281/bundles/core/core.entry.js:1:275664

In parallel with that, i also encounter other "authentication" errors? when i restart elasticsearch in debug logs such as below;

[2024-02-26T10:10:31,001][INFO ][o.e.x.t.t.TransformFailureHandler] [elasticsearch] [slo-summary-timeslices-weekly-aligned] Transform encountered an exception: [Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.]; Will automatically retry [1/-1]
org.elasticsearch.action.search.SearchPhaseExecutionException:
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:709) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:456) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:220) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1165) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:913) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$10(TransportSearchAction.java:337) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:449) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:304) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:114) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:623) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:602) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:505) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1028) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:994) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:401) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:310) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:230) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:133) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:108) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1074) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:381) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:151) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:292) ~[?:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:147) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:262) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:108) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:86) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:54) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:59) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:317) ~[?:?]
	at org.elasticsearch.xpack.transform.checkpoint.TimeBasedCheckpointProvider.sourceHasChanged(TimeBasedCheckpointProvider.java:80) ~[?:?]
	at org.elasticsearch.xpack.transform.transforms.TransformIndexer.onStart(TransformIndexer.java:374) ~[?:?]
	at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.lambda$maybeTriggerAsyncJob$5(AsyncTwoPhaseIndexer.java:230) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) ~[elasticsearch-8.12.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:230) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:451) ~[elasticsearch-8.12.2.jar:?]
	... 103 more
[2024-02-26T10:10:31,917][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] successfully loaded geoip database file [GeoLite2-City.mmdb]
[2024-02-26T10:10:38,834][INFO ][o.e.c.r.a.AllocationService] [elasticsearch] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.internal.alerts-observability.uptime.alerts-default-000001][0]]])." previous.health="RED" reason="shards started [[.internal.alerts-observability.uptime.alerts-default-000001][0]]"

Also there was this error when i boot up, than i updated elasticsearch with yum update and error gone. How can i validate if my installment go corrupt or not?

Feb 20 12:48:43 elasticsearch systemd-entrypoint[1502]: ERROR: Found no [elastic-apm-agent] jar under [/usr/share/elasticsearch/modules/apm]! Installation is corrupt., with exit code 70
Feb 20 12:48:43 elasticsearch systemd[1]: elasticsearch.service: main process exited, code=exited, status=70/n/a
Feb 20 12:48:43 elasticsearch systemd[1]: Failed to start Elasticsearch.
Feb 20 12:48:43 elasticsearch systemd[1]: Unit elasticsearch.service entered failed state.
Feb 20 12:48:43 elasticsearch systemd[1]: elasticsearch.service failed.
Feb 20 12:58:54 elasticsearch systemd[1]: Starting Elasticsearch...

[FALEN (Ali Can Saykal)]

Also, please see below kibana logs that shows errors such as "Session is no longer available and cannot be re-authenticated."

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:08.989+01:00","message":"Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_arm64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":3254,"uptime":34.388880215},"trace":{"id":"224f9bf15de21045df4c5256f2363346"},"transaction":{"id":"3d3ad03b7cc6b96f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:13.525+01:00","message":"Logging in with provider \"basic\" (basic)","log":{"level":"INFO","logger":"plugins.security.routes"},"process":{"pid":3254,"uptime":38.925389617},"trace":{"id":"b6970ddd1ab7b93a23746840c7093198"},"transaction":{"id":"7c0cd47cfc85789d"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.803+01:00","message":"Session is no longer available and cannot be re-authenticated.","log":{"level":"WARN","logger":"plugins.security.authenticator"},"process":{"pid":3254,"uptime":41.202807919},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.803+01:00","message":"Re-authentication cannot be handled.","log":{"level":"ERROR","logger":"plugins.security.authentication"},"process":{"pid":3254,"uptime":41.203169623},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.811+01:00","message":"Session is no longer available and cannot be re-authenticated.","log":{"level":"WARN","logger":"plugins.security.authenticator"},"process":{"pid":3254,"uptime":41.211372104},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.812+01:00","message":"Re-authentication cannot be handled.","log":{"level":"ERROR","logger":"plugins.security.authentication"},"process":{"pid":3254,"uptime":41.211668096},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"http":{"response":{"status_code":401},"request":{"method":"post","path":"/api/core/capabilities"}},"error":{"message":"security_exception\n\tRoot causes:\n\t\tsecurity_exception: missing authentication credentials for REST request [/_security/user/_has_privileges]"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.813+01:00","message":"401 Unauthorized","log":{"level":"ERROR","logger":"http"},"process":{"pid":3254,"uptime":41.212627435},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:45.316+01:00","message":"Logging in with provider \"basic\" (basic)","log":{"level":"INFO","logger":"plugins.security.routes"},"process":{"pid":3254,"uptime":70.715600459},"trace":{"id":"7a367a1814d33aef0351f7d1c0399624"},"transaction":{"id":"0ae553d79777f397"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:53:07.396+01:00","message":"Started. Checking for changes to endpoint artifacts","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":92.795610345},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"9cb8bf3ebfe61c95"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:53:07.403+01:00","message":"Last computed manifest not available yet","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":92.803040961},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"9cb8bf3ebfe61c95"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:53:07.404+01:00","message":"Complete. Task run took 8ms [ stated: 2024-02-26T09:53:07.396Z ]","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":92.804061362},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"9cb8bf3ebfe61c95"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:54:10.392+01:00","message":"Started. Checking for changes to endpoint artifacts","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":155.793901486},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"b32f4e7bb72b6da0"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:54:10.409+01:00","message":"Last computed manifest not available yet","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":155.808561862},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"b32f4e7bb72b6da0"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:54:10.409+01:00","message":"Complete. Task run took 17ms [ stated: 2024-02-26T09:54:10.392Z ]","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":155.808970252},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"b32f4e7bb72b6da0"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:54:16.390+01:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":3254,"uptime":161.789882778},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"0471f4277c2a37b7"}}

also below is taken from elasticsearch logs, that is same time perioud

[2024-02-26T10:52:15,593][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] pre-authorizing child action [indices:data/read/search[phase/query]] of parent action [indices:data/read/search]
[2024-02-26T10:52:15,593][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,595][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,596][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,599][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,600][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,610][DEBUG][o.e.x.m.e.l.LocalExporter] [elasticsearch] monitoring index templates are installed on master node, service can start
[2024-02-26T10:52:15,798][DEBUG][o.e.x.s.a.AuthenticatorChain] [elasticsearch] No valid credentials found in request [rest request uri [/_security/user/_has_privileges]], rejecting
[2024-02-26T10:52:15,799][DEBUG][r.suppressed             ] [elasticsearch] path: /_security/user/_has_privileges, params: {}, status: 401
[2024-02-26T10:52:15,807][DEBUG][o.e.x.s.a.AuthenticatorChain] [elasticsearch] No valid credentials found in request [rest request uri [/_security/user/_has_privileges]], rejecting
[2024-02-26T10:52:15,808][DEBUG][r.suppressed             ] [elasticsearch] path: /_security/user/_has_privileges, params: {}, status: 401

Next step, enabled below

PUT _cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.security.authc": "trace"
  }
}

There isnt any trace log related to authentication error, you can below see the moment it accept the request, than it rejects, than i see kibana tries enrollment token. Can there be a problem with Kibana enrollment?
[2024-02-26T11:09:20,807][TRACE][o.e.x.s.a.s.ServiceAccountService] [elasticsearch] attempt to authenticate service account token [elastic/kibana/enroll-process-token-1691932966864]

[2024-02-26T11:09:18,549][TRACE][o.e.x.s.a.AuthenticatorChain] [elasticsearch] Established authentication [Authentication[effectiveSubject=Subject{version=8560001, user=User[username=username,roles=[superuser,kibana_admin],fullName=****,email=*****,metadata={}], realm={Realm[native.default_native] on Node[elasticsearch]}, type=USER, metadata={}},type=REALM]] for request [rest request uri [/_security/user/_has_privileges]]
[2024-02-26T11:09:20,787][DEBUG][o.e.x.s.a.AuthenticatorChain] [elasticsearch] No valid credentials found in request [rest request uri [/_security/user/_has_privileges]], rejecting
[2024-02-26T11:09:20,787][DEBUG][r.suppressed             ] [elasticsearch] path: /_security/user/_has_privileges, params: {}, status: 401
org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for REST request [/_security/user/_has_privileges]
[2024-02-26T11:09:20,801][DEBUG][o.e.x.s.a.AuthenticatorChain] [elasticsearch] No valid credentials found in request [rest request uri [/_security/user/_has_privileges]], rejecting
[2024-02-26T11:09:20,801][DEBUG][r.suppressed             ] [elasticsearch] path: /_security/user/_has_privileges, params: {}, status: 401
org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for REST request [/_security/user/_has_privileges]
[2024-02-26T11:09:32,175][TRACE][o.e.x.s.a.AuthenticatorChain] [elasticsearch] Established authentication [Authentication[effectiveSubject=Subject{version=8560001, user=User[username=*****,roles=[],fullName=****l,email=*****,metadata={}], realm={Realm[_es_api_key._es_api_key] on Node[elasticsearch]}, type=API_KEY, metadata={_security_api_key_creator_realm_name=default_native, _security_api_key_limited_by_role_descriptors=org.elasticsearch.common.bytes.BytesArray@aae36c38, _security_api_key_id=idGe0IoBEEbOGtPV5G8K, _security_api_key_type=rest, _security_api_key_creator_realm_type=native, _security_api_key_name=Alerting: monitoring_alert_disk_usage/Elasticsearch Disk Usage, _security_api_key_role_descriptors=org.elasticsearch.common.bytes.BytesArray@1323}},type=API_KEY]] for request [rest request uri [/_security/user/_has_privileges]]

[FALEN (Ali Can Saykal)]

Update!

I tried it with Brave, Chrome, Safari

Safari seems to be working fine without kicking me out. So i think its an frontend issue with chromium browsers

Update2!

Also works fine if you left Discover screen with 1 second auto refresh active on any browser, so it might be not related to safari. More about refresh token getting invalidated?

[tsullivan (Tim Sullivan)]

Can you share your kibana.yml? Maybe you are missing the xpack.security.encryptionKey setting, which allows you to carry over authenticated sessions after Elasticsearch restarts. Are there warning messages in the startup logs of the Kibana server to that effect?

Or you might be using the Token Service for authentication in Elasticsearch, and need a longer timeout limit. The default is 20 minutes. See the details of xpack.security.authc.token.timeout in Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/8.12/security-settings.html#token-service-settings

[FALEN (Ali Can Saykal)]

Hi Tim,
below is current kibana conf

server.host: "172.xx.xx.xx"
server.publicBaseUrl: "https://xxxx.com"
server.name: "xxx-elasticsearch"

logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid

elasticsearch.hosts: ['https://172.xx.xx.xx:9200']
elasticsearch.serviceAccountToken: xxxxxx
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1691932967567.crt]
xpack.encryptedSavedObjects.encryptionKey: xxx 32char random numbers
xpack.security.audit.enabled: true

"xpack.security.encryptionKey" is missing, added it but same problem still occurs

"xpack.security.authc.token.timeout" also configured this to 60m in elasticsearch.yml

Only error im seeing in kibana logs while startup is

"message":"Failed to resolve ELSER model definition: Error: Platinum, Enterprise or trial license needed","log":{"level":"ERROR","logger":"plugins.observabilityAIAssistant"},"process":{"pid":8323,"uptime":32.386862025},"trace":{"id":"802f128626c20a75e87766d4624116ed"},"transaction":{"id":"38f171b0b3f4e7d1"}}

This node was previously licensed, now its switched to basic

Additionally, can it be a Proxy issue? There are lot of errors in browser logs when logout occurs

[tsullivan (Tim Sullivan)]

@FALEN

"xpack.security.encryptionKey" is missing, added it but same problem still occurs

I wouldn't take this to mean that this setting isn't important. It is important for consistency in user sessions over cluster restarts.

Additionally:

1. Check that your license hasn't become expired.
2. The browser logs in Kibana tend to be noisy, and some of those look like ones I see during normal use cases, except for the Uncaught exceptions. But those are likely due to lack of exception handling in the code, when the exception is that authentication has been lost.
3. This can certainly be a proxy issue. Check that the proxy rules are not mistakenly stripping out headers from the forwarded requests.