I am running a 3 node cluster, but last week i was constantly getting the kibana server not ready error, but on looking at it, my kibana_system user was not able to authenticate to the cluster neither any of my users created during setup are able to authenticate. I am getting a lot of authentication error per minute. Initially i was getting a corrupted traslog error and not able to restore the cluster state error. the translog file in one of the indices in one of my drive. I had IO error in that specific traslog folder location, so i used fsck to clear the issues. that drive had around 600GB indices in it. What should i do, the logs in the cluster are crucial. How can i retrieve it. Please Help.
[2024-06-21T12:07:14,961][WARN ][o.e.x.s.a.ApiKeyAuthenticator] [node1] Authentication using apikey failed - apikey authentication for id EVdXGowBSOBwmarphxO4 encountered a failure
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.checkGlobalBlock(TransportSingleShardAction.java:112) ~[elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:146) ~[elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.<init>(TransportSingleShardAction.java:130) ~[elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:98) ~[elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:51) ~[elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:186) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:190) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:574) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:568) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:572) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:568) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:26) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:572) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:568) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:103) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:572) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:568) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:100) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:572) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:568) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:87) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:572) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:568) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:85) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:26) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:572) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:568) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:128) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:568) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:553) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:447) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:964) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:928) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:388) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:41) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:1012) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:380) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:440) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:377) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:262) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:148) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:253) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:154) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:137) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:264) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:103) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:172) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$1(SecurityActionFilter.java:120) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.core.security.SecurityContext.executeAsUser(SecurityContext.java:140) [x-pack-core-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:118) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:117) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:161) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:512) [elasticsearch-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:127) [x-pack-core-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.ApiKeyService.loadApiKeyAndValidateCredentials(ApiKeyService.java:539) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.ApiKeyService.tryAuthenticate(ApiKeyService.java:483) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator.authenticate(ApiKeyAuthenticator.java:50) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:172) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:166) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:166) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:144) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:105) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:150) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:128) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$30(Security.java:1529) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.xpack.security.Security.lambda$getHttpServerTransportWithHeadersValidator$36(Security.java:1619) [x-pack-security-7.17.21.jar:7.17.21]
at org.elasticsearch.http.netty4.internal.HttpHeadersAuthenticatorUtils.lambda$getValidatorInboundHandler$2(HttpHeadersAuthenticatorUtils.java:48) [transport-netty4-client-7.17.21.jar:7.17.21]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.requestStart(Netty4HttpHeaderValidator.java:126) [transport-netty4-client-7.17.21.jar:7.17.21]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:61) [transport-netty4-client-7.17.21.jar:7.17.21]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.lang.Thread.run(Thread.java:1583) [?:?]
The API Key failure, and the error message are a symptom, not a cause.
This node (and possibly the whole cluster) is unhealthy - or at least it has not yet fully started up.
You'll need to look for other log messages that explain why that is the case.