Error logstash [logstash.outputs.elasticsearch] Encountered a retryable error code=>503

San9 · March 23, 2023, 9:34am

Hi all. I'm a beginner at this.
When setting up another pipelayer, after starting it, the following errors started to appear in the log for all other pipelines:

logstash[363391]: [2023-03-23T08:05:49,424][ERROR][logstash.outputs.elasticsearch][logs-pipe][610eb6a905ab6cf9cb6040ab7070702a938ccdb3359f5bae29458cad64c65be3] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>2167}

 logstash[363391]: [2023-03-23T08:05:47,525][ERROR][logstash.outputs.elasticsearch][cvp-pipe][1b2be7beffad378f6f35630fffe400b42b5c7ca8f80a792fdc5eebe84b46b13e] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>417}

logs on elasticserch (ingest):

[2023-03-23T08:19:17,966][WARN ][r.suppressed             ] [ingest02.test] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:163) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:60) ~[?:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:35) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:398) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:945) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:909) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:412) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:171) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:402) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:378) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$2(AuthorizationService.java:263) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:139) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:185) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:47) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:283) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:200) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:175) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:136) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:265) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
		 at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:165) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$prepareRequest$0(RestMonitoringBulkAction.java:102) [x-pack-monitoring-8.3.3.jar:8.3.3]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:112) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:106) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:415) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:349) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:325) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:203) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:122) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:189) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:217) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:162) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-8.3.3.jar:?]
		at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:146) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:82) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:181) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:137) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:96) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:100) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:389) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:468) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:304) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:371) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:450) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:345) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:98) [transport-netty4-8.3.3.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:88) [transport-netty4-8.3.3.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
		 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.76.Final.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.76.Final.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-03-23T08:19:18,947][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-03-23T08:19:22,850][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
		[2023-03-23T08:19:25,214][INFO ][o.e.t.TcpTransport       ] [ingest02.test] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/ingest02.test:45522, remoteAddress=master01.test/master01.test:9301, profile=default}], disconnecting from relevant node: Connection timed out
[2023-03-23T08:19:25,215][INFO ][o.e.t.ClusterConnectionManager] [ingest02.test] transport connection to [{master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}] closed by remote
[2023-03-23T08:19:25,215][INFO ][o.e.c.c.JoinHelper       ] [ingest02.test] failed to join {master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true} with JoinRequest{sourceNode={ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}{xpack.installed=true}, minimumTerm=23, optionalJoin=Optional[Join{term=23, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}{xpack.installed=true}, targetNode={master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true}}]}
org.elasticsearch.transport.NodeDisconnectedException: [master01.test][master01.test:9301][internal:cluster/coordination/join] disconnected
[2023-03-23T08:19:25,731][WARN ][o.e.c.c.JoinHelper       ] [ingest02.test] last failed join attempt was 515ms ago, failed to join {master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true} with JoinRequest{sourceNode={ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}{xpack.installed=true}, minimumTerm=23, optionalJoin=Optional[Join{term=23, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}{xpack.installed=true}, targetNode={master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true}}]}
org.elasticsearch.transport.NodeDisconnectedException: [master01.test][master01.test:9301][internal:cluster/coordination/join] disconnected
[2023-03-23T08:19:25,731][WARN ][o.e.c.c.ClusterFormationFailureHelper] [ingest02.test] master not discovered yet: have discovered [{ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}, {master02.test}{8FaQnG_TT6uDoxRMB_8duw}{i9mUjuDTSgCLvOFbijnKxw}{master02.test}{master02.test}{master02.test:9301}{m}, {master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}, {v-elk-em03.test}{xKXXQhmVT9efjGWQKQ-iGw}{-NvIYjg-TsOvHorkU5IssQ}{v-elk-em03.test}{master03.test}{master03.test:9301}{m}]; discovery will continue using [master01.test:9301, master02.test:9301, master03.test:9301] from hosts providers and [{master02.test}{8FaQnG_TT6uDoxRMB_8duw}{i9mUjuDTSgCLvOFbijnKxw}{master02.test}{master02.test}{master02.test:9301}{m}, {master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}, {v-elk-em03.test}{xKXXQhmVT9efjGWQKQ-iGw}{-NvIYjg-TsOvHorkU5IssQ}{v-elk-em03.test}{master03.test}{master03.test:9301}{m}] from last-known cluster state; node term 23, last-accepted version 183559 in term 23
[2023-03-23T08:19:25,876][INFO ][o.e.c.s.ClusterApplierService] [ingest02.test] master node changed {previous [], current [{master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}]}, term: 23, version: 183560, reason: ApplyCommitRequest{term=23, version=183560, sourceNode={master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true}}

errors disappear only after restarting logstash.
how to solve this problem?

San9 · April 6, 2023, 12:28pm

hi, any ideas?

leandrojmp · April 6, 2023, 1:16pm

San9:

[2023-03-23T08:19:17,966][WARN ][r.suppressed             ] [ingest02.test] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];

This error means that your Elasticsearch cluster was not working because there was no master node available.

This would make Elasticsearch reject requests from Logstash, which will give you the error you got on Logstash.

You need to troubleshoot what happend to your master.

Also, restarting Logstash should have no impact in Elasticsearch.

Are you running both in the same machine?

San9 · April 6, 2023, 2:06pm

my model: 3 - node master, 2 - logstash, 4 - node date, 2 - ingest. This error occurs periodically, I can not understand the pattern of this event

Dasher · April 26, 2023, 8:44am

Can you try with the elasticsearch url only i.e https://ingest02.test:9200

San9 · May 5, 2023, 1:54pm

errors on logstash continue, there are even more of them.

Apr 26 07:56:08 logstash02.test logstash[85183]: [2023-04-26T07:56:08,339][ERROR][logstash.outputs.elasticsearch][pexip-syslog-pipe][4f51eaf43f3c46bf49092ff4a1140431d5fd15e2c9c8c44e1af06b102bd6825d] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>1004}
Apr 26 07:56:08 logstash02.test logstash[85183]: [2023-04-26T07:56:08,387][ERROR][logstash.outputs.elasticsearch][sap-pipe][af5fe57bbbddf233cad08f208390566cdf8b7c661cd9e7dafa35385525eb5c6a] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>80524}
Apr 26 07:56:08 logstash02.test logstash[85183]: [2023-04-26T07:56:08,607][ERROR][logstash.outputs.elasticsearch][vcs-logs-pipe][610eb6a905ab6cf9cb6040ab7070702a938ccdb3359f5bae29458cad64c65be3] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>4506}

and

Apr 26 07:56:09 logstash02.test logstash[85183]: [2023-04-26T07:56:09,046][ERROR][logstash.outputs.elasticsearch][k2-sharepoint-metric-pipe] Unable to get license information {:url=>"https://user:xxxxxx@ingest02.test:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '503' contacting Elasticsearch at URL 'https://ingest02.test:9200/_license'"}
Apr 26 07:56:09 logstash02.test logstash[85183]: [2023-04-26T07:56:09,046][ERROR][logstash.outputs.elasticsearch][k2-sharepoint-metric-pipe] Could not connect to a compatible version of Elasticsearch {:url=>"https://user:xxxxxx@ingest02.test:9200/"}

why might these problems be?

San9 · May 5, 2023, 1:55pm

do you mean use one node?

rachel_gomez · May 16, 2023, 4:50am

The error message you provided indicates that Logstash is encountering a retryable error with a status code of 503 when trying to communicate with Elasticsearch. A status code of 503 typically indicates that the Elasticsearch service is temporarily unavailable or overloaded.

Here are a few steps you can take to troubleshoot and resolve this issue:

Verify Elasticsearch availability: Ensure that Elasticsearch is up and running and accessible from the machine running Logstash. Check if there are any issues with Elasticsearch itself, such as high resource usage or connectivity problems.

Check Elasticsearch logs: Inspect the Elasticsearch logs to see if there are any errors or warnings that could provide insights into the issue. Look for any indications of service unavailability or resource limitations.

Monitor system resources: Check the system resources (CPU, memory, disk space) on the machine running Logstash and Elasticsearch. Make sure that there are no resource constraints that could impact their performance.

Adjust Logstash pipeline configuration: Review your Logstash configuration file (usually logstash.conf) and ensure that it is correctly set up for sending data to Elasticsearch. Pay attention to the Elasticsearch output plugin settings, including the Elasticsearch host, port, and any authentication credentials.

Increase Logstash output plugin retry settings: In your Logstash configuration, you can try increasing the number of retry attempts and the interval between retries for the Elasticsearch output plugin. This can be done by modifying the retry_max_interval and retry_max_items parameters. However, keep in mind that this is a temporary workaround and may not address the underlying issue.

Scale Elasticsearch infrastructure: If you frequently encounter 503 errors and your Elasticsearch cluster is under heavy load, consider scaling your Elasticsearch infrastructure by adding more nodes or increasing the resources allocated to the existing nodes. This can help distribute the load and improve the availability of Elasticsearch.

Enable Elasticsearch slow log: Enabling the Elasticsearch slow log can provide additional insights into any performance issues or slow queries that might be causing the 503 errors. Analyzing the slow log can help identify bottlenecks or problematic queries that need optimization.

Consider network issues: Check for any network-related issues between Logstash and Elasticsearch, such as firewalls, proxies, or network restrictions that could be interfering with the communication. Ensure that the required ports are open and that there are no connectivity problems.

If the issue persists, it may be helpful to reach out to the Logstash or Elasticsearch communities or consult with the support teams for these tools. They can provide more specific guidance based on your setup and configuration.

When seeking support, provide relevant details such as your Logstash configuration, Elasticsearch version, and any error logs or messages. This information will assist the support team or community members in understanding the problem and offering appropriate solutions.

I hope these suggestions help you troubleshoot and resolve the issue with Logstash encountering a retryable error code 503 when communicating with Elasticsearch.

Regards,
Rachel Gomez

San9 · May 26, 2023, 10:05am

Thank you for your recommendations.
I checked for network problems and possible blockages, but I did not find any.
I checked whether the elastic service is running or stopped on problem nodes - the service is running.
But in the logs on ingests, I see such elastic logs:

[2023-05-26T09:54:19,676][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-05-26T09:54:19,678][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-05-26T09:54:20,413][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-05-26T09:54:21,432][WARN ][r.suppressed             ] [ingest02.test] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:163) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:60) ~[?:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:35) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:398) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:945) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:909) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:412) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:171) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:402) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:378) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$2(AuthorizationService.java:263) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:139) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:185) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:47) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:283) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:200) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:175) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:136) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:265) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:165) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$prepareRequest$0(RestMonitoringBulkAction.java:102) [x-pack-monitoring-8.3.3.jar:8.3.3]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:112) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:106) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:415) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:349) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:325) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:203) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:122) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:189) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:217) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:162) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:146) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:82) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:181) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:137) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:96) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:100) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:389) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:468) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:304) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:371) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:450) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:345) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:98) [transport-netty4-8.3.3.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:88) [transport-netty4-8.3.3.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.76.Final.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.76.Final.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-05-26T09:54:21,978][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

what do you mean by "Enable Elasticsearch slow log" ?

leandrojmp · May 26, 2023, 12:12pm

San9:

[2023-05-26T09:54:19,676][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master]

Your cluster is not working, it says that there is no master node for the cluster, you need to troubleshoot why there is no master in your cluster, check the logs of your master node.

The fact that the service is running does not mean it is working, you need to check the health of the cluster.

adimi_worou · June 7, 2023, 7:48am

Hi, i’ve the same problem. Logstash can’t load data from mysql db to elasticsearch.

I use docker.

Thanks for your help

San9 · June 30, 2023, 9:30am

I won’t say anything about the docket, since I haven’t used it yet.

San9 · June 30, 2023, 9:33am

Thank you very much for the hint, then I will look at the logs on the masters in more detail. I will report changes.

San9 · July 26, 2023, 2:56pm

I looked at the logs on the masters (I have 3 of them), this is what I saw:
files master01 - elk-prod.log

[2023-07-26T11:37:18,858][INFO ][o.e.t.TcpTransport       ] [master01] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/1.1.5.210:9301, remoteAddress=/1.1.6.14:52044, profile=default}], disconnecting from relevant node: Connection reset

files master02 - elk-prod.log

[2023-07-26T00:01:12,684][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,723][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,759][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,795][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,832][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,866][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,901][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,938][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,976][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:13,020][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]

files master03 - elk-prod.log

[2023-07-26T11:32:53,206][INFO ][o.e.t.TcpTransport       ] [master03] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/1.1.5.212:9301, remoteAddress=/1.1.6.14:50736, profile=default}], disconnecting from relevant node: Connection reset

Or do I need to look at some other files with logs? I just didn’t understand that deeply and never had such problems before.
Thanks

system · August 23, 2023, 2:57pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash 7.10 Error Logstash	3	275	January 14, 2021
Logstash output elasticsearch Error code 504 Logstash	2	2115	June 6, 2019
Require suggestion for the logstash processing : Logstash Logstash	1	465	November 16, 2018
Logstash failing with "Encountered a retryable error." Logstash	3	7502	January 10, 2018
Logstash Error - elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://elastichost:9200/_bulk"} Logstash	2	304	May 30, 2021

Error logstash [logstash.outputs.elasticsearch] Encountered a retryable error code=>503

Related topics