Randomly encountered auth errors in Kibana severity increased

FALEN · February 26, 2024, 9:14am

Hi Community,

I'm randomly encountering auth errors as shared below, severity of these errors are increased and now i cannot stay logged in for 1-2 minutes before encountering one

security_exception Root causes: security_exception: missing authentication credentials for REST request [/_security/user/_has_privileges]

Version: 8.12.2
Build: 70281
Error
    at fetch_Fetch.fetchResponse (https://***.com/70281/bundles/core/core.entry.js:1:277774)
    at async https://***.com/70281/bundles/core/core.entry.js:1:275707
    at async https://***.com/70281/bundles/core/core.entry.js:1:275664

In parallel with that, i also encounter other "authentication" errors? when i restart elasticsearch in debug logs such as below;

[2024-02-26T10:10:31,001][INFO ][o.e.x.t.t.TransformFailureHandler] [elasticsearch] [slo-summary-timeslices-weekly-aligned] Transform encountered an exception: [Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.]; Will automatically retry [1/-1]
org.elasticsearch.action.search.SearchPhaseExecutionException:
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:709) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:456) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:220) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1165) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:913) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$10(TransportSearchAction.java:337) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:449) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:304) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:114) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:623) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:617) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:602) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:505) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1028) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:994) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:401) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:310) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:230) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:133) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:108) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1074) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:381) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:151) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:292) ~[?:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:147) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:262) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:108) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:86) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:54) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:59) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:317) ~[?:?]
	at org.elasticsearch.xpack.transform.checkpoint.TimeBasedCheckpointProvider.sourceHasChanged(TimeBasedCheckpointProvider.java:80) ~[?:?]
	at org.elasticsearch.xpack.transform.transforms.TransformIndexer.onStart(TransformIndexer.java:374) ~[?:?]
	at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.lambda$maybeTriggerAsyncJob$5(AsyncTwoPhaseIndexer.java:230) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) ~[elasticsearch-8.12.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at org.elasticsearch.action.search.SearchPhase.doCheckNoMissingShards(SearchPhase.java:61) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:230) ~[elasticsearch-8.12.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:451) ~[elasticsearch-8.12.2.jar:?]
	... 103 more
[2024-02-26T10:10:31,917][INFO ][o.e.i.g.DatabaseNodeService] [elasticsearch] successfully loaded geoip database file [GeoLite2-City.mmdb]
[2024-02-26T10:10:38,834][INFO ][o.e.c.r.a.AllocationService] [elasticsearch] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.internal.alerts-observability.uptime.alerts-default-000001][0]]])." previous.health="RED" reason="shards started [[.internal.alerts-observability.uptime.alerts-default-000001][0]]"

Also there was this error when i boot up, than i updated elasticsearch with yum update and error gone. How can i validate if my installment go corrupt or not?

Feb 20 12:48:43 elasticsearch systemd-entrypoint[1502]: ERROR: Found no [elastic-apm-agent] jar under [/usr/share/elasticsearch/modules/apm]! Installation is corrupt., with exit code 70
Feb 20 12:48:43 elasticsearch systemd[1]: elasticsearch.service: main process exited, code=exited, status=70/n/a
Feb 20 12:48:43 elasticsearch systemd[1]: Failed to start Elasticsearch.
Feb 20 12:48:43 elasticsearch systemd[1]: Unit elasticsearch.service entered failed state.
Feb 20 12:48:43 elasticsearch systemd[1]: elasticsearch.service failed.
Feb 20 12:58:54 elasticsearch systemd[1]: Starting Elasticsearch...

FALEN · February 26, 2024, 9:55am

Also, please see below kibana logs that shows errors such as "Session is no longer available and cannot be re-authenticated."

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:08.989+01:00","message":"Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_arm64/headless_shell","log":{"level":"INFO","logger":"plugins.screenshotting.chromium"},"process":{"pid":3254,"uptime":34.388880215},"trace":{"id":"224f9bf15de21045df4c5256f2363346"},"transaction":{"id":"3d3ad03b7cc6b96f"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:13.525+01:00","message":"Logging in with provider \"basic\" (basic)","log":{"level":"INFO","logger":"plugins.security.routes"},"process":{"pid":3254,"uptime":38.925389617},"trace":{"id":"b6970ddd1ab7b93a23746840c7093198"},"transaction":{"id":"7c0cd47cfc85789d"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.803+01:00","message":"Session is no longer available and cannot be re-authenticated.","log":{"level":"WARN","logger":"plugins.security.authenticator"},"process":{"pid":3254,"uptime":41.202807919},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.803+01:00","message":"Re-authentication cannot be handled.","log":{"level":"ERROR","logger":"plugins.security.authentication"},"process":{"pid":3254,"uptime":41.203169623},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.811+01:00","message":"Session is no longer available and cannot be re-authenticated.","log":{"level":"WARN","logger":"plugins.security.authenticator"},"process":{"pid":3254,"uptime":41.211372104},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.812+01:00","message":"Re-authentication cannot be handled.","log":{"level":"ERROR","logger":"plugins.security.authentication"},"process":{"pid":3254,"uptime":41.211668096},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"http":{"response":{"status_code":401},"request":{"method":"post","path":"/api/core/capabilities"}},"error":{"message":"security_exception\n\tRoot causes:\n\t\tsecurity_exception: missing authentication credentials for REST request [/_security/user/_has_privileges]"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:15.813+01:00","message":"401 Unauthorized","log":{"level":"ERROR","logger":"http"},"process":{"pid":3254,"uptime":41.212627435},"trace":{"id":"5a5fd3db86ff3b04fdd7b6b56d436640"},"transaction":{"id":"068d803f5f15c30e"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:52:45.316+01:00","message":"Logging in with provider \"basic\" (basic)","log":{"level":"INFO","logger":"plugins.security.routes"},"process":{"pid":3254,"uptime":70.715600459},"trace":{"id":"7a367a1814d33aef0351f7d1c0399624"},"transaction":{"id":"0ae553d79777f397"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:53:07.396+01:00","message":"Started. Checking for changes to endpoint artifacts","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":92.795610345},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"9cb8bf3ebfe61c95"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:53:07.403+01:00","message":"Last computed manifest not available yet","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":92.803040961},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"9cb8bf3ebfe61c95"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:53:07.404+01:00","message":"Complete. Task run took 8ms [ stated: 2024-02-26T09:53:07.396Z ]","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":92.804061362},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"9cb8bf3ebfe61c95"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:54:10.392+01:00","message":"Started. Checking for changes to endpoint artifacts","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":155.793901486},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"b32f4e7bb72b6da0"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:54:10.409+01:00","message":"Last computed manifest not available yet","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":155.808561862},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"b32f4e7bb72b6da0"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:54:10.409+01:00","message":"Complete. Task run took 17ms [ stated: 2024-02-26T09:54:10.392Z ]","log":{"level":"INFO","logger":"plugins.securitySolution.endpoint:user-artifact-packager:1.0.0"},"process":{"pid":3254,"uptime":155.808970252},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"b32f4e7bb72b6da0"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-02-26T10:54:16.390+01:00","message":"Fleet Usage: {\"agents_enabled\":true,\"agents\":{\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"inactive\":0,\"unenrolled\":0,\"total_all_statuses\":0,\"updating\":0},\"fleet_server\":{\"total_all_statuses\":0,\"total_enrolled\":0,\"healthy\":0,\"unhealthy\":0,\"offline\":0,\"updating\":0,\"num_host_urls\":0}}","log":{"level":"INFO","logger":"plugins.fleet"},"process":{"pid":3254,"uptime":161.789882778},"trace":{"id":"ace2e0555b5b4fbbe4eac4a02250ff2a"},"transaction":{"id":"0471f4277c2a37b7"}}

also below is taken from elasticsearch logs, that is same time perioud

[2024-02-26T10:52:15,593][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] pre-authorizing child action [indices:data/read/search[phase/query]] of parent action [indices:data/read/search]
[2024-02-26T10:52:15,593][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,595][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,596][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,599][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,600][DEBUG][o.e.x.s.a.PreAuthorizationUtils] [elasticsearch] adding authorization for parent action [indices:data/read/search] to the thread context
[2024-02-26T10:52:15,610][DEBUG][o.e.x.m.e.l.LocalExporter] [elasticsearch] monitoring index templates are installed on master node, service can start
[2024-02-26T10:52:15,798][DEBUG][o.e.x.s.a.AuthenticatorChain] [elasticsearch] No valid credentials found in request [rest request uri [/_security/user/_has_privileges]], rejecting
[2024-02-26T10:52:15,799][DEBUG][r.suppressed             ] [elasticsearch] path: /_security/user/_has_privileges, params: {}, status: 401
[2024-02-26T10:52:15,807][DEBUG][o.e.x.s.a.AuthenticatorChain] [elasticsearch] No valid credentials found in request [rest request uri [/_security/user/_has_privileges]], rejecting
[2024-02-26T10:52:15,808][DEBUG][r.suppressed             ] [elasticsearch] path: /_security/user/_has_privileges, params: {}, status: 401

Next step, enabled below

PUT _cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.security.authc": "trace"
  }
}

There isnt any trace log related to authentication error, you can below see the moment it accept the request, than it rejects, than i see kibana tries enrollment token. Can there be a problem with Kibana enrollment?
[2024-02-26T11:09:20,807][TRACE][o.e.x.s.a.s.ServiceAccountService] [elasticsearch] attempt to authenticate service account token [elastic/kibana/enroll-process-token-1691932966864]

[2024-02-26T11:09:18,549][TRACE][o.e.x.s.a.AuthenticatorChain] [elasticsearch] Established authentication [Authentication[effectiveSubject=Subject{version=8560001, user=User[username=username,roles=[superuser,kibana_admin],fullName=****,email=*****,metadata={}], realm={Realm[native.default_native] on Node[elasticsearch]}, type=USER, metadata={}},type=REALM]] for request [rest request uri [/_security/user/_has_privileges]]
[2024-02-26T11:09:20,787][DEBUG][o.e.x.s.a.AuthenticatorChain] [elasticsearch] No valid credentials found in request [rest request uri [/_security/user/_has_privileges]], rejecting
[2024-02-26T11:09:20,787][DEBUG][r.suppressed             ] [elasticsearch] path: /_security/user/_has_privileges, params: {}, status: 401
org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for REST request [/_security/user/_has_privileges]
[2024-02-26T11:09:20,801][DEBUG][o.e.x.s.a.AuthenticatorChain] [elasticsearch] No valid credentials found in request [rest request uri [/_security/user/_has_privileges]], rejecting
[2024-02-26T11:09:20,801][DEBUG][r.suppressed             ] [elasticsearch] path: /_security/user/_has_privileges, params: {}, status: 401
org.elasticsearch.ElasticsearchSecurityException: missing authentication credentials for REST request [/_security/user/_has_privileges]
[2024-02-26T11:09:32,175][TRACE][o.e.x.s.a.AuthenticatorChain] [elasticsearch] Established authentication [Authentication[effectiveSubject=Subject{version=8560001, user=User[username=*****,roles=[],fullName=****l,email=*****,metadata={}], realm={Realm[_es_api_key._es_api_key] on Node[elasticsearch]}, type=API_KEY, metadata={_security_api_key_creator_realm_name=default_native, _security_api_key_limited_by_role_descriptors=org.elasticsearch.common.bytes.BytesArray@aae36c38, _security_api_key_id=idGe0IoBEEbOGtPV5G8K, _security_api_key_type=rest, _security_api_key_creator_realm_type=native, _security_api_key_name=Alerting: monitoring_alert_disk_usage/Elasticsearch Disk Usage, _security_api_key_role_descriptors=org.elasticsearch.common.bytes.BytesArray@1323}},type=API_KEY]] for request [rest request uri [/_security/user/_has_privileges]]

FALEN · February 26, 2024, 11:01am

Update!

I tried it with Brave, Chrome, Safari

Safari seems to be working fine without kicking me out. So i think its an frontend issue with chromium browsers

Update2!

Also works fine if you left Discover screen with 1 second auto refresh active on any browser, so it might be not related to safari. More about refresh token getting invalidated?

tsullivan · March 19, 2024, 10:10pm

Can you share your kibana.yml? Maybe you are missing the xpack.security.encryptionKey setting, which allows you to carry over authenticated sessions after Elasticsearch restarts. Are there warning messages in the startup logs of the Kibana server to that effect?

Or you might be using the Token Service for authentication in Elasticsearch, and need a longer timeout limit. The default is 20 minutes. See the details of xpack.security.authc.token.timeout in Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/8.12/security-settings.html#token-service-settings

FALEN · March 20, 2024, 11:07am

Hi Tim,
below is current kibana conf

server.host: "172.xx.xx.xx"
server.publicBaseUrl: "https://xxxx.com"
server.name: "xxx-elasticsearch"

logging:
  appenders:
    file:
      type: file
      fileName: /var/log/kibana/kibana.log
      layout:
        type: json
  root:
    appenders:
      - default
      - file
pid.file: /run/kibana/kibana.pid

elasticsearch.hosts: ['https://172.xx.xx.xx:9200']
elasticsearch.serviceAccountToken: xxxxxx
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1691932967567.crt]
xpack.encryptedSavedObjects.encryptionKey: xxx 32char random numbers
xpack.security.audit.enabled: true

"xpack.security.encryptionKey" is missing, added it but same problem still occurs

"xpack.security.authc.token.timeout" also configured this to 60m in elasticsearch.yml

Only error im seeing in kibana logs while startup is

"message":"Failed to resolve ELSER model definition: Error: Platinum, Enterprise or trial license needed","log":{"level":"ERROR","logger":"plugins.observabilityAIAssistant"},"process":{"pid":8323,"uptime":32.386862025},"trace":{"id":"802f128626c20a75e87766d4624116ed"},"transaction":{"id":"38f171b0b3f4e7d1"}}

This node was previously licensed, now its switched to basic

Additionally, can it be a Proxy issue? There are lot of errors in browser logs when logout occurs

tsullivan · March 20, 2024, 2:53pm

I wouldn't take this to mean that this setting isn't important. It is important for consistency in user sessions over cluster restarts.

Additionally:

Check that your license hasn't become expired.
The browser logs in Kibana tend to be noisy, and some of those look like ones I see during normal use cases, except for the Uncaught exceptions. But those are likely due to lack of exception handling in the code, when the exception is that authentication has been lost.
This can certainly be a proxy issue. Check that the proxy rules are not mistakenly stripping out headers from the forwarded requests.

system · April 17, 2024, 2:53pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.