Issues restarting kibana

elaair · July 6, 2024, 10:55pm

I forgot to power off the computer running elastic and kibana prior to turning off the house power. When I turned it back on, logstash and elastic were fine but kibana is complaining about some migration error. This setup had been running for several months so I don't understand the migration error.

[2024-07-06T17:44:35.323-05:00][INFO ][savedobjects-service] [.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 124012ms.
[2024-07-06T17:46:39.335-05:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with '[index_not_yellow_timeout] Timeout waiting for the status of the [.kibana_task_manager_8.12.1_001] index to become 'yellow' Refer to https://www.elastic.co/guide/en/kibana/8.12/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 11 in 64 seconds.
[2024-07-06T17:46:39.336-05:00][INFO ][savedobjects-service] [.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> WAIT_FOR_YELLOW_SOURCE. took: 124013ms.

From the elastic log:

[2024-07-07T09:56:28,567][WARN ][r.suppressed             ] [schroew-ASB20] path: /.kibana_task_manager/_search, params: {index=.kibana_task_manager}, status: 503org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.server@8.14.2/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)
	at org.elasticsearch.server@8.14.2/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:404)
	at org.elasticsearch.server@8.14.2/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)
	at org.elasticsearch.server@8.14.2/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497)
	at org.elasticsearch.server@8.14.2/org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:312)

Rios · July 7, 2024, 4:47am

Check ES log, might be more info what is happening.
Is there more traces in Ki log?

elaair · July 7, 2024, 3:51pm

An interesting point, I can GET all of those .kibana indexes that kibana is complaining about so they seem to be there..

Rios · July 7, 2024, 5:02pm

And? Is there any traces in logs?

elaair · July 7, 2024, 5:44pm

It seems that this index is red:

".kibana_task_manager_8.12.1_001" : {
      "status" : "red",
      "number_of_shards" : 1,
      "number_of_replicas" : 0,
      "active_primary_shards" : 0,
      "active_shards" : 0,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 1,
      "shards" : {
        "0" : {
          "status" : "red",
          "primary_active" : false,
          "active_shards" : 0,
          "relocating_shards" : 0,
          "initializing_shards" : 0,
          "unassigned_shards" : 1
        }
      }
    },

elaair · July 7, 2024, 6:12pm

The elastic logs has lots of this:

[2024-07-07T13:10:49,005][WARN ][r.suppressed             ] [schroew-ASB20] path: /.kibana_task_manager/_search, params: {index=.kibana_task_manager}, status: 503
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:404) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:312) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:248) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:223) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1175) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:934) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:353) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:465) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:309) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:113) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:96) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:182) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:217) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:637) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:635) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.ValidateRequestInterceptor.intercept(ValidateRequestInterceptor.java:20) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:635) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:635) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:635) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:635) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:635) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:635) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:635) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:631) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:616) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:514) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1030) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:996) ~[?:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:420) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:382) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:302) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:205) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:170) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1076) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:388) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:507) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:439) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:326) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:154) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:193) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:211) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:291) ~[?:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:209) ~[?:?]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:186) ~[?:?]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:150) ~[?:?]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:342) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$5(SecurityActionFilter.java:178) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:264) ~[?:?]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:174) ~[?:?]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:131) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:93) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:68) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:105) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:81) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:357) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:125) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:106) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.rest.RestController$1.onResponse(RestController.java:452) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.rest.RestController$1.onResponse(RestController.java:446) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:89) ~[?:?]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$intercept$0(SecurityRestFilter.java:81) ~[?:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:171) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109) ~[?:?]
	at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90) ~[?:?]
	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.intercept(SecurityRestFilter.java:75) ~[?:?]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:446) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:606) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:329) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:487) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:583) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:460) ~[elasticsearch-8.14.2.jar:?]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:126) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:116) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:209) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:152) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator$1.lambda$onResponse$0(Netty4HttpHeaderValidator.java:125) ~[?:?]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[?:?]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: org.elasticsearch.action.NoShardAvailableActionException

elaair · July 7, 2024, 6:16pm

The start of the kibana log:

[2024-07-07T12:48:22.525-05:00][INFO ][plugins.assetManager] Server is NOT enabled
[2024-07-07T12:48:23.472-05:00][INFO ][elasticsearch-service] Successfully connected to Elasticsearch after waiting for 102 milliseconds
[2024-07-07T12:48:23.664-05:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2024-07-07T12:48:23.666-05:00][INFO ][savedobjects-service] Starting saved objects migrations
[2024-07-07T12:48:23.811-05:00][INFO ][savedobjects-service] [.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 95ms.
[2024-07-07T12:48:23.822-05:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 98ms.
[2024-07-07T12:48:23.830-05:00][INFO ][savedobjects-service] [.kibana] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 19ms.
[2024-07-07T12:48:23.867-05:00][INFO ][savedobjects-service] [.kibana_ingest] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 92ms.
[2024-07-07T12:48:23.889-05:00][INFO ][savedobjects-service] [.kibana_security_solution] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 135ms.
[2024-07-07T12:48:23.997-05:00][INFO ][savedobjects-service] [.kibana_alerting_cases] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 232ms.
[2024-07-07T12:48:24.025-05:00][INFO ][savedobjects-service] [.kibana_ingest] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 158ms.
[2024-07-07T12:48:24.058-05:00][INFO ][savedobjects-service] [.kibana_analytics] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 320ms.
[2024-07-07T12:48:24.137-05:00][INFO ][savedobjects-service] [.kibana_alerting_cases] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 140ms.
[2024-07-07T12:48:24.153-05:00][INFO ][savedobjects-service] [.kibana_alerting_cases] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> CLEANUP_UNKNOWN_AND_EXCLUDED. took: 16ms.
[2024-07-07T12:48:24.220-05:00][INFO ][savedobjects-service] [.kibana_security_solution] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 331ms.
[2024-07-07T12:48:24.278-05:00][INFO ][savedobjects-service] [.kibana] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> CLEANUP_UNKNOWN_AND_EXCLUDED. took: 448ms.
[2024-07-07T12:48:24.383-05:00][INFO ][savedobjects-service] [.kibana_analytics] WAIT_FOR_YELLOW_SOURCE -> UPDATE_SOURCE_MAPPINGS_PROPERTIES. took: 325ms.
[2024-07-07T12:48:24.445-05:00][INFO ][savedobjects-service] [.kibana_ingest] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> CLEANUP_UNKNOWN_AND_EXCLUDED. took: 419ms.
[2024-07-07T12:48:24.709-05:00][INFO ][savedobjects-service] [.kibana_analytics] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> CLEANUP_UNKNOWN_AND_EXCLUDED. took: 326ms.
[2024-07-07T12:48:24.745-05:00][INFO ][savedobjects-service] [.kibana_security_solution] UPDATE_SOURCE_MAPPINGS_PROPERTIES -> CLEANUP_UNKNOWN_AND_EXCLUDED. took: 525ms.
[2024-07-07T12:48:24.873-05:00][ERROR][savedobjects-service] [.kibana] Action failed with 'search_phase_execution_exception
	Root causes:
		no_shard_available_action_exception: null'. Retrying attempt 1 in 2 seconds.
[2024-07-07T12:48:24.874-05:00][INFO ][savedobjects-service] [.kibana] CLEANUP_UNKNOWN_AND_EXCLUDED -> CLEANUP_UNKNOWN_AND_EXCLUDED. took: 595ms.
[2024-07-07T12:48:24.914-05:00][ERROR][savedobjects-service] [.kibana_ingest] Action failed with 'search_phase_execution_exception

Rios · July 8, 2024, 3:33am

There is an error in ES, somewhere.
Does ES API work?
Is there more details in ES logs?
How did you get error? Suddenly or after upgrade from older version or something else?
Do you have enough disk space?
Do you use xpack for an authentication integration?

Topic		Replies	Views
Fail to upgrade Kibana from 7.11 to 7.12 Kibana	10	2880	May 5, 2021
[.kibana_task_manager] Action failed with 'Request timed out' Kibana	3	333	September 5, 2023
.kibana index failed Kibana	9	912	November 18, 2022
[error][savedobjects-service] [.kibana_task_manager] Action failed with 'Request timed out'. [savedobjects-service] [.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_INDEX_BULK -> REINDEX_SOURCE_TO_TEMP_INDEX_BULK Kibana	9	3536	August 29, 2022
[.kibana_task_manager] Action failed with 'Request timed out' Kibana	4	3884	November 8, 2021

Issues restarting kibana

Related topics