Error logstash [logstash.outputs.elasticsearch] Encountered a retryable error code=>503

Elastic Stack Logstash
1

Hi all. I'm a beginner at this.
When setting up another pipelayer, after starting it, the following errors started to appear in the log for all other pipelines:

logstash[363391]: [2023-03-23T08:05:49,424][ERROR][logstash.outputs.elasticsearch][logs-pipe][610eb6a905ab6cf9cb6040ab7070702a938ccdb3359f5bae29458cad64c65be3] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>2167}

 logstash[363391]: [2023-03-23T08:05:47,525][ERROR][logstash.outputs.elasticsearch][cvp-pipe][1b2be7beffad378f6f35630fffe400b42b5c7ca8f80a792fdc5eebe84b46b13e] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>417}

logs on elasticserch (ingest):

[2023-03-23T08:19:17,966][WARN ][r.suppressed             ] [ingest02.test] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:163) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:60) ~[?:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:35) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:398) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:945) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:909) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:412) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:171) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:402) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:378) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$2(AuthorizationService.java:263) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:139) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:185) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:47) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:283) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:200) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:175) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:136) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:265) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
		 at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:165) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$prepareRequest$0(RestMonitoringBulkAction.java:102) [x-pack-monitoring-8.3.3.jar:8.3.3]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:112) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:106) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:415) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:349) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:325) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:203) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:122) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:189) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:217) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:162) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-8.3.3.jar:?]
		at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:146) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:82) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:181) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:137) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:96) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:100) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:389) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:468) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:304) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:371) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:450) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:345) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:98) [transport-netty4-8.3.3.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:88) [transport-netty4-8.3.3.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
		 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.76.Final.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.76.Final.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-03-23T08:19:18,947][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-03-23T08:19:22,850][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
		[2023-03-23T08:19:25,214][INFO ][o.e.t.TcpTransport       ] [ingest02.test] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/ingest02.test:45522, remoteAddress=master01.test/master01.test:9301, profile=default}], disconnecting from relevant node: Connection timed out
[2023-03-23T08:19:25,215][INFO ][o.e.t.ClusterConnectionManager] [ingest02.test] transport connection to [{master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}] closed by remote
[2023-03-23T08:19:25,215][INFO ][o.e.c.c.JoinHelper       ] [ingest02.test] failed to join {master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true} with JoinRequest{sourceNode={ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}{xpack.installed=true}, minimumTerm=23, optionalJoin=Optional[Join{term=23, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}{xpack.installed=true}, targetNode={master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true}}]}
org.elasticsearch.transport.NodeDisconnectedException: [master01.test][master01.test:9301][internal:cluster/coordination/join] disconnected
[2023-03-23T08:19:25,731][WARN ][o.e.c.c.JoinHelper       ] [ingest02.test] last failed join attempt was 515ms ago, failed to join {master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true} with JoinRequest{sourceNode={ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}{xpack.installed=true}, minimumTerm=23, optionalJoin=Optional[Join{term=23, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}{xpack.installed=true}, targetNode={master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true}}]}
org.elasticsearch.transport.NodeDisconnectedException: [master01.test][master01.test:9301][internal:cluster/coordination/join] disconnected
[2023-03-23T08:19:25,731][WARN ][o.e.c.c.ClusterFormationFailureHelper] [ingest02.test] master not discovered yet: have discovered [{ingest02.test}{cV10VSK1STS60hWeyGqtcQ}{_A2KzWmkTjWAir5L4gGF6A}{ingest02.test}{ingest02.test}{ingest02.test:9301}{i}, {master02.test}{8FaQnG_TT6uDoxRMB_8duw}{i9mUjuDTSgCLvOFbijnKxw}{master02.test}{master02.test}{master02.test:9301}{m}, {master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}, {v-elk-em03.test}{xKXXQhmVT9efjGWQKQ-iGw}{-NvIYjg-TsOvHorkU5IssQ}{v-elk-em03.test}{master03.test}{master03.test:9301}{m}]; discovery will continue using [master01.test:9301, master02.test:9301, master03.test:9301] from hosts providers and [{master02.test}{8FaQnG_TT6uDoxRMB_8duw}{i9mUjuDTSgCLvOFbijnKxw}{master02.test}{master02.test}{master02.test:9301}{m}, {master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}, {v-elk-em03.test}{xKXXQhmVT9efjGWQKQ-iGw}{-NvIYjg-TsOvHorkU5IssQ}{v-elk-em03.test}{master03.test}{master03.test:9301}{m}] from last-known cluster state; node term 23, last-accepted version 183559 in term 23
[2023-03-23T08:19:25,876][INFO ][o.e.c.s.ClusterApplierService] [ingest02.test] master node changed {previous [], current [{master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}]}, term: 23, version: 183560, reason: ApplyCommitRequest{term=23, version=183560, sourceNode={master01.test}{J5e2urQZT0enBQgEtqxXSA}{5wcG18TETcKd0fDe8l6rGQ}{master01.test}{master01.test}{master01.test:9301}{m}{xpack.installed=true}}

errors disappear only after restarting logstash.
how to solve this problem? :sleepy: :tired_face:

2

hi, any ideas?

3

This error means that your Elasticsearch cluster was not working because there was no master node available.

This would make Elasticsearch reject requests from Logstash, which will give you the error you got on Logstash.

You need to troubleshoot what happend to your master.

Also, restarting Logstash should have no impact in Elasticsearch.

Are you running both in the same machine?

4

my model: 3 - node master, 2 - logstash, 4 - node date, 2 - ingest. This error occurs periodically, I can not understand the pattern of this event

6

Can you try with the elasticsearch url only i.e https://ingest02.test:9200

7

errors on logstash continue, there are even more of them.

Apr 26 07:56:08 logstash02.test logstash[85183]: [2023-04-26T07:56:08,339][ERROR][logstash.outputs.elasticsearch][pexip-syslog-pipe][4f51eaf43f3c46bf49092ff4a1140431d5fd15e2c9c8c44e1af06b102bd6825d] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>1004}
Apr 26 07:56:08 logstash02.test logstash[85183]: [2023-04-26T07:56:08,387][ERROR][logstash.outputs.elasticsearch][sap-pipe][af5fe57bbbddf233cad08f208390566cdf8b7c661cd9e7dafa35385525eb5c6a] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>80524}
Apr 26 07:56:08 logstash02.test logstash[85183]: [2023-04-26T07:56:08,607][ERROR][logstash.outputs.elasticsearch][vcs-logs-pipe][610eb6a905ab6cf9cb6040ab7070702a938ccdb3359f5bae29458cad64c65be3] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://ingest02.test:9200/_bulk", :content_length=>4506}

and

Apr 26 07:56:09 logstash02.test logstash[85183]: [2023-04-26T07:56:09,046][ERROR][logstash.outputs.elasticsearch][k2-sharepoint-metric-pipe] Unable to get license information {:url=>"https://user:xxxxxx@ingest02.test:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '503' contacting Elasticsearch at URL 'https://ingest02.test:9200/_license'"}
Apr 26 07:56:09 logstash02.test logstash[85183]: [2023-04-26T07:56:09,046][ERROR][logstash.outputs.elasticsearch][k2-sharepoint-metric-pipe] Could not connect to a compatible version of Elasticsearch {:url=>"https://user:xxxxxx@ingest02.test:9200/"}

why might these problems be?

8

do you mean use one node?

9

The error message you provided indicates that Logstash is encountering a retryable error with a status code of 503 when trying to communicate with Elasticsearch. A status code of 503 typically indicates that the Elasticsearch service is temporarily unavailable or overloaded.

Here are a few steps you can take to troubleshoot and resolve this issue:

Verify Elasticsearch availability: Ensure that Elasticsearch is up and running and accessible from the machine running Logstash. Check if there are any issues with Elasticsearch itself, such as high resource usage or connectivity problems.

Check Elasticsearch logs: Inspect the Elasticsearch logs to see if there are any errors or warnings that could provide insights into the issue. Look for any indications of service unavailability or resource limitations.

Monitor system resources: Check the system resources (CPU, memory, disk space) on the machine running Logstash and Elasticsearch. Make sure that there are no resource constraints that could impact their performance.

Adjust Logstash pipeline configuration: Review your Logstash configuration file (usually logstash.conf) and ensure that it is correctly set up for sending data to Elasticsearch. Pay attention to the Elasticsearch output plugin settings, including the Elasticsearch host, port, and any authentication credentials.

Increase Logstash output plugin retry settings: In your Logstash configuration, you can try increasing the number of retry attempts and the interval between retries for the Elasticsearch output plugin. This can be done by modifying the retry_max_interval and retry_max_items parameters. However, keep in mind that this is a temporary workaround and may not address the underlying issue.

Scale Elasticsearch infrastructure: If you frequently encounter 503 errors and your Elasticsearch cluster is under heavy load, consider scaling your Elasticsearch infrastructure by adding more nodes or increasing the resources allocated to the existing nodes. This can help distribute the load and improve the availability of Elasticsearch.

Enable Elasticsearch slow log: Enabling the Elasticsearch slow log can provide additional insights into any performance issues or slow queries that might be causing the 503 errors. Analyzing the slow log can help identify bottlenecks or problematic queries that need optimization.

Consider network issues: Check for any network-related issues between Logstash and Elasticsearch, such as firewalls, proxies, or network restrictions that could be interfering with the communication. Ensure that the required ports are open and that there are no connectivity problems.

If the issue persists, it may be helpful to reach out to the Logstash or Elasticsearch communities or consult with the support teams for these tools. They can provide more specific guidance based on your setup and configuration.

When seeking support, provide relevant details such as your Logstash configuration, Elasticsearch version, and any error logs or messages. This information will assist the support team or community members in understanding the problem and offering appropriate solutions.

I hope these suggestions help you troubleshoot and resolve the issue with Logstash encountering a retryable error code 503 when communicating with Elasticsearch.

Regards,
Rachel Gomez

1 Like
10

Thank you for your recommendations.
I checked for network problems and possible blockages, but I did not find any.
I checked whether the elastic service is running or stopped on problem nodes - the service is running.
But in the logs on ingests, I see such elastic logs:

[2023-05-26T09:54:19,676][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-05-26T09:54:19,678][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-05-26T09:54:20,413][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-05-26T09:54:21,432][WARN ][r.suppressed             ] [ingest02.test] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:163) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:60) ~[?:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:35) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:398) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:945) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:909) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:412) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:171) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:402) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:378) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$2(AuthorizationService.java:263) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:139) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:185) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:47) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:55) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:283) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:50) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:200) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:175) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:136) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:265) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:165) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$prepareRequest$0(RestMonitoringBulkAction.java:102) [x-pack-monitoring-8.3.3.jar:8.3.3]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:112) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$2(SecondaryAuthenticator.java:84) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:94) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:78) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$2(SecurityRestFilter.java:106) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:415) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.writeAuthToContext(AuthenticatorChain.java:349) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.finishAuthentication(AuthenticatorChain.java:325) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.maybeLookupRunAsUser(AuthenticatorChain.java:203) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$doAuthenticate$1(AuthenticatorChain.java:122) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:189) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$3(RealmsAuthenticator.java:217) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:141) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$0(RealmsAuthenticator.java:162) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$1(CachingUsernamePasswordRealm.java:155) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.handleCachedAuthentication(CachingUsernamePasswordRealm.java:242) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:139) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:134) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$2(RealmsAuthenticator.java:146) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:233) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:82) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:181) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:159) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) [x-pack-core-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:137) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:96) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:149) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:127) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:100) [x-pack-security-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:389) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:468) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:304) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:371) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:450) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:345) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:98) [transport-netty4-8.3.3.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:88) [transport-netty4-8.3.3.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) [netty-handler-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.76.Final.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.76.Final.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.76.Final.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-05-26T09:54:21,978][WARN ][r.suppressed             ] [ingest02.test] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:177) ~[elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:622) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:495) [elasticsearch-8.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) [elasticsearch-8.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

what do you mean by "Enable Elasticsearch slow log" ?

11

Your cluster is not working, it says that there is no master node for the cluster, you need to troubleshoot why there is no master in your cluster, check the logs of your master node.

The fact that the service is running does not mean it is working, you need to check the health of the cluster.

12

Hi, i’ve the same problem. Logstash can’t load data from mysql db to elasticsearch.

I use docker.

Thanks for your help

13

I won’t say anything about the docket, since I haven’t used it yet.

14

Thank you very much for the hint, then I will look at the logs on the masters in more detail. I will report changes.

15

I looked at the logs on the masters (I have 3 of them), this is what I saw:
files master01 - elk-prod.log

[2023-07-26T11:37:18,858][INFO ][o.e.t.TcpTransport       ] [master01] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/1.1.5.210:9301, remoteAddress=/1.1.6.14:52044, profile=default}], disconnecting from relevant node: Connection reset

files master02 - elk-prod.log

[2023-07-26T00:01:12,684][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,723][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,759][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,795][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,832][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,866][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,901][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,938][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:12,976][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]
[2023-07-26T00:01:13,020][INFO ][o.e.c.m.MetadataMappingService] [master02] [aws-s3-s3-2023.07.26/9HF6_LbyTFWazeaX6h06DQ] update_mapping [_doc]

files master03 - elk-prod.log

[2023-07-26T11:32:53,206][INFO ][o.e.t.TcpTransport       ] [master03] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/1.1.5.212:9301, remoteAddress=/1.1.6.14:50736, profile=default}], disconnecting from relevant node: Connection reset

Or do I need to look at some other files with logs? I just didn’t understand that deeply and never had such problems before.
Thanks

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.