Elasticsearch "close connection exception caught on transport layer ... disconnecting from relevant node: Connection reset

San9 · May 8, 2024, 9:16am

Unstable operation of the elastic cluster. In the logs I see that the ingest nodes (I have two of them) are disconnected from the master in a chaotic manner after a certain time. Because of this problem, I can't transfer data from Lostash. I collected the logs, but I cannot correctly diagnose the cause of the problem.
on ingest logs :

[2024-05-08T07:04:47,471][WARN ][r.suppressed             ] [ingest01.my] path: /_bulk, params: {}, status: 503
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction.ensureClusterStateThenForkAndExecute(TransportBulkAction.java:259) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:249) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:94) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:623) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:602) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:505) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1028) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:994) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:312) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:151) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:244) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:147) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:262) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:108) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:86) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.bulk(AbstractClient.java:461) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:98) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:94) ~[?:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:85) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109) ~[?:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90) ~[?:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79) ~[?:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:441) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:570) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:325) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:72) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-05-08T07:04:47,797][INFO ][o.e.t.TcpTransport       ] [ingest01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.6.13:48452, remoteAddress=10.1.5.210/10.1.5.210:9301, profile=default}], disconnecting from relevant node: Connection timed out
[2024-05-08T07:04:47,797][INFO ][o.e.t.ClusterConnectionManager] [ingest01.my] transport connection to [{master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}] closed by remote
[2024-05-08T07:04:47,797][INFO ][o.e.c.c.JoinHelper       ] [ingest01.my] failed to join {master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0} with JoinRequest{sourceNode={ingest01.my}{pZK4-zUxTzGbfKIcLFVkwg}{7VXOReW2S6ujMopuD9Subw}{ingest01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}, compatibilityVersions=CompatibilityVersions[transportVersion=8560001, systemIndexMappingsVersion={.triggered_watches=MappingsVersion[version=1, hash=-502826165], .secrets-inference=MappingsVersion[version=1, hash=-1434574148], .fleet-agents-7=MappingsVersion[version=1, hash=1021268528], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .ml-config=MappingsVersion[version=1, hash=1455005892], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=2, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-1=MappingsVersion[version=1, hash=1647955624], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-840564854], .fleet-actions-7=MappingsVersion[version=1, hash=961753072], .tasks=MappingsVersion[version=0, hash=-945584329], .ml-meta=MappingsVersion[version=1, hash=976416370], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .security-7=MappingsVersion[version=1, hash=-1061511639], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=3, hash=919553140], .inference=MappingsVersion[version=1, hash=645017346], .async-search=MappingsVersion[version=0, hash=-1403744380], .fleet-policies-7=MappingsVersion[version=1, hash=762674907], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], features=[features_supported, usage.data_tiers.precalculate_stats, health.dsl.info], minimumTerm=79, optionalJoin=Optional[Join[votingNode={ingest01.my}{pZK4-zUxTzGbfKIcLFVkwg}{7VXOReW2S6ujMopuD9Subw}{ingest01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}, masterCandidateNode={master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}, term=79, lastAcceptedTerm=0, lastAcceptedVersion=0]]}
org.elasticsearch.transport.NodeDisconnectedException: [master01.my][10.1.5.210:9301][internal:cluster/coordination/join] disconnected
[2024-05-08T07:04:48,268][WARN ][r.suppressed             ] [ingest01.my] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}, status: 503
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:174) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:61) ~[?:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:36) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:455) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1028) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:994) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:469) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:186) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:459) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:151) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:292) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:147) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:262) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:108) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:86) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$prepareRequest$0(RestMonitoringBulkAction.java:101) ~[?:?]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:94) ~[?:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:85) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109) ~[?:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90) ~[?:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79) ~[?:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:441) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:570) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:325) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120) ~[?:?]
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[?:?]
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106) ~[?:?]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[?:?]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-05-08T07:04:48,500][INFO ][o.e.c.s.ClusterApplierService] [ingest01.my] master node changed {previous [], current [{master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}]}, term: 79, version: 1659837, reason: ApplyCommitRequest{term=79, version=1659837, sourceNode={master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}}

on master node logs:

[2024-05-08T05:42:40,612][INFO ][o.e.t.TcpTransport       ] [master01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.210:9301, remoteAddress=/10.1.6.13:39520, profile=default}], disconnecting from relevant node: Connection reset
[2024-05-08T07:04:48,540][INFO ][o.e.c.c.NodeJoinExecutor ] [master01.my] node-join: [{ingest01.my}{pZK4-zUxTzGbfKIcLFVkwg}{7VXOReW2S6ujMopuD9Subw}{ingest01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.12.2}{7000099-8500010}] with reason [rejoining]
[2024-05-08T07:09:47,109][INFO ][o.e.t.TcpTransport       ] [master01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.210:9301, remoteAddress=/10.1.6.13:48452, profile=default}], disconnecting from relevant node: Connection reset

Subrahmanyam_Veerank · May 8, 2024, 9:21am

Share the TLS related configuration for further digging.

San9 · May 8, 2024, 9:59am

and now I see such failures in attempts to connect from Ingest to the data node:

[2024-05-08T09:49:13,028][WARN ][o.e.t.OutboundHandler    ] [ingest01.my] sending transport message [Request{indices:data/write/bulk[s]}{36925995}{false}{true}{false}] of size [2450] on [Netty4TcpChannel{localAddress=/10.1.6.13:56102, remoteAddress=10.1.5.208/10.1.5.208:9301, profile=default}] took [7279ms] which is above the warn threshold of [5000ms] with success [false]

and there are many such attempts...

Subrahmanyam_Veerank · May 8, 2024, 11:45am

Check whether nodes are reachable with each other and ensure no network issues between them.

curl -X GET -u elastic: https://xxxx:9200 --cacert /etc/elasticsearch/certs/<certificate_authorities>

San9 · May 8, 2024, 11:51am

Hi, Here is my elastiсearch configuration on the ingest node:

bootstrap.memory_lock: true
cluster.name: my-elk
discovery.seed_hosts:
- master01.my:9301
- master02.my:9301
- master03.my:9301
http.port: 9200
network.host: 0
node.roles:
- ingest
- transform
transport.port: 9301
xpack.monitoring.collection.enabled: true
node.name: ingest01.my
#################################### Paths ####################################
# Path to directory containing configuration (this file and logging.yml):
path.repo: /mnt/nfs-share
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
action.auto_create_index: true
xpack.security.enabled: true
xpack.security.transport.ssl:
  enabled: true
  verification_mode: "certificate"
  keystore.path: "/etc/elasticsearch/certs/elastic-certificates.p12"
  truststore.path: "/etc/elasticsearch/certs/elastic-certificates.p12"
xpack.security.http.ssl:
  enabled: true
  keystore.path: "/etc/elasticsearch/certs/elastic-certificates.p12"
#truststore.path: "/etc/elasticsearch/certs/elastic-certificates.p12"
xpack.security.http.ssl.supported_protocols: [ "TLSv1.3", "TLSv1.2" ]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
transport.host: 0.0.0.0
  #transport.ping_schedule: 1800s

if I misunderstood you, please correct me.

My template logstash output:

output {
  elasticsearch {
        hosts => ["https://ingest01.my:9200", "https://ingest02.my:9200"]
          index => "index-%{+YYYY.MM}"
          user => "username"
          password => "password"
          ssl => true
          cacert => "/usr/local/share/ca-certificates/elastic-stack-ca-pub-prod.crt"
          ssl_certificate_verification => true
         }
		 }

The data is being transferred, then problems arise with it, I can’t understand what the reason is.

Thanks

San9 · May 8, 2024, 12:12pm

To check I usually use:

curl -X GET -u elastic:'mypassword' https://master01.my:9200
{
  "name" : "master01.my",
  "cluster_name" : "my-elk",
  "cluster_uuid" : "ACjIGY_AT-6yr0tCdySHqw",
  "version" : {
    "number" : "8.12.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "48a287ab9497e852de30327444b0809e55d46466",
    "build_date" : "2024-02-19T10:04:32.774273190Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

this time it passed, but when there are problems then it gives a 503 error

San9 · May 14, 2024, 8:54am

Does anyone have any ideas about this problem? I noticed that the problem is with ingests. This does not happen simultaneously; first, one node may be unavailable (the master cannot be seen), and then after a while the second.