Elasticsearch "close connection exception caught on transport layer ... disconnecting from relevant node: Connection reset

Elastic Stack Elasticsearch
1

Unstable operation of the elastic cluster. In the logs I see that the ingest nodes (I have two of them) are disconnected from the master in a chaotic manner after a certain time. Because of this problem, I can't transfer data from Lostash. I collected the logs, but I cannot correctly diagnose the cause of the problem.
on ingest logs :

[2024-05-08T07:04:47,471][WARN ][r.suppressed             ] [ingest01.my] path: /_bulk, params: {}, status: 503
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction.ensureClusterStateThenForkAndExecute(TransportBulkAction.java:259) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:249) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:94) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:623) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:27) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:124) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.ShardSearchRequestInterceptor.intercept(ShardSearchRequestInterceptor.java:24) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:79) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:21) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.DlsFlsLicenseRequestInterceptor.intercept(DlsFlsLicenseRequestInterceptor.java:106) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:98) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:85) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:621) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$1.onResponse(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestCacheDisablingInterceptor.intercept(SearchRequestCacheDisablingInterceptor.java:53) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:617) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:602) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$13(AuthorizationService.java:505) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1028) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:994) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:312) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:498) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:151) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:244) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:147) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:262) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:108) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:86) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.bulk(AbstractClient.java:461) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:98) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:94) ~[?:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:85) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109) ~[?:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90) ~[?:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79) ~[?:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:441) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:570) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:325) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.channelRead(Netty4HttpHeaderValidator.java:72) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-05-08T07:04:47,797][INFO ][o.e.t.TcpTransport       ] [ingest01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.6.13:48452, remoteAddress=10.1.5.210/10.1.5.210:9301, profile=default}], disconnecting from relevant node: Connection timed out
[2024-05-08T07:04:47,797][INFO ][o.e.t.ClusterConnectionManager] [ingest01.my] transport connection to [{master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}] closed by remote
[2024-05-08T07:04:47,797][INFO ][o.e.c.c.JoinHelper       ] [ingest01.my] failed to join {master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0} with JoinRequest{sourceNode={ingest01.my}{pZK4-zUxTzGbfKIcLFVkwg}{7VXOReW2S6ujMopuD9Subw}{ingest01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}, compatibilityVersions=CompatibilityVersions[transportVersion=8560001, systemIndexMappingsVersion={.triggered_watches=MappingsVersion[version=1, hash=-502826165], .secrets-inference=MappingsVersion[version=1, hash=-1434574148], .fleet-agents-7=MappingsVersion[version=1, hash=1021268528], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .ml-config=MappingsVersion[version=1, hash=1455005892], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=2, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-1=MappingsVersion[version=1, hash=1647955624], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-840564854], .fleet-actions-7=MappingsVersion[version=1, hash=961753072], .tasks=MappingsVersion[version=0, hash=-945584329], .ml-meta=MappingsVersion[version=1, hash=976416370], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .security-7=MappingsVersion[version=1, hash=-1061511639], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=3, hash=919553140], .inference=MappingsVersion[version=1, hash=645017346], .async-search=MappingsVersion[version=0, hash=-1403744380], .fleet-policies-7=MappingsVersion[version=1, hash=762674907], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], features=[features_supported, usage.data_tiers.precalculate_stats, health.dsl.info], minimumTerm=79, optionalJoin=Optional[Join[votingNode={ingest01.my}{pZK4-zUxTzGbfKIcLFVkwg}{7VXOReW2S6ujMopuD9Subw}{ingest01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}, masterCandidateNode={master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}, term=79, lastAcceptedTerm=0, lastAcceptedVersion=0]]}
org.elasticsearch.transport.NodeDisconnectedException: [master01.my][10.1.5.210:9301][internal:cluster/coordination/join] disconnected
[2024-05-08T07:04:48,268][WARN ][r.suppressed             ] [ingest01.my] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}, status: 503
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:174) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:61) ~[?:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:36) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:87) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:455) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1028) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:994) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:469) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:186) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:459) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:151) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:292) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
        at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:147) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:93) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:262) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:85) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:62) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:196) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:108) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:86) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:381) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$prepareRequest$0(RestMonitoringBulkAction.java:101) ~[?:?]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:94) ~[?:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:85) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109) ~[?:?]
        at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90) ~[?:?]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:79) ~[?:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:441) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:570) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:325) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:458) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:554) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:431) ~[elasticsearch-8.12.2.jar:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120) ~[?:?]
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[?:?]
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106) ~[?:?]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[?:?]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-05-08T07:04:48,500][INFO ][o.e.c.s.ClusterApplierService] [ingest01.my] master node changed {previous [], current [{master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}]}, term: 79, version: 1659837, reason: ApplyCommitRequest{term=79, version=1659837, sourceNode={master01.my}{J5e2urQZT0enBQgEtqxXSA}{VNBNGS-lRz2BQsUvUUKVdw}{master01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0}}

on master node logs:

[2024-05-08T05:42:40,612][INFO ][o.e.t.TcpTransport       ] [master01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.210:9301, remoteAddress=/10.1.6.13:39520, profile=default}], disconnecting from relevant node: Connection reset
[2024-05-08T07:04:48,540][INFO ][o.e.c.c.NodeJoinExecutor ] [master01.my] node-join: [{ingest01.my}{pZK4-zUxTzGbfKIcLFVkwg}{7VXOReW2S6ujMopuD9Subw}{ingest01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.12.2}{7000099-8500010}] with reason [rejoining]
[2024-05-08T07:09:47,109][INFO ][o.e.t.TcpTransport       ] [master01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.210:9301, remoteAddress=/10.1.6.13:48452, profile=default}], disconnecting from relevant node: Connection reset
2

Share the TLS related configuration for further digging.

3

and now I see such failures in attempts to connect from Ingest to the data node:

[2024-05-08T09:49:13,028][WARN ][o.e.t.OutboundHandler    ] [ingest01.my] sending transport message [Request{indices:data/write/bulk[s]}{36925995}{false}{true}{false}] of size [2450] on [Netty4TcpChannel{localAddress=/10.1.6.13:56102, remoteAddress=10.1.5.208/10.1.5.208:9301, profile=default}] took [7279ms] which is above the warn threshold of [5000ms] with success [false]

and there are many such attempts...

4

Check whether nodes are reachable with each other and ensure no network issues between them.

curl -X GET -u elastic: https://xxxx:9200 --cacert /etc/elasticsearch/certs/<certificate_authorities>

5

Hi, Here is my elastiсearch configuration on the ingest node:

bootstrap.memory_lock: true
cluster.name: my-elk
discovery.seed_hosts:
- master01.my:9301
- master02.my:9301
- master03.my:9301
http.port: 9200
network.host: 0
node.roles:
- ingest
- transform
transport.port: 9301
xpack.monitoring.collection.enabled: true
node.name: ingest01.my
#################################### Paths ####################################
# Path to directory containing configuration (this file and logging.yml):
path.repo: /mnt/nfs-share
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
action.auto_create_index: true
xpack.security.enabled: true
xpack.security.transport.ssl:
  enabled: true
  verification_mode: "certificate"
  keystore.path: "/etc/elasticsearch/certs/elastic-certificates.p12"
  truststore.path: "/etc/elasticsearch/certs/elastic-certificates.p12"
xpack.security.http.ssl:
  enabled: true
  keystore.path: "/etc/elasticsearch/certs/elastic-certificates.p12"
#truststore.path: "/etc/elasticsearch/certs/elastic-certificates.p12"
xpack.security.http.ssl.supported_protocols: [ "TLSv1.3", "TLSv1.2" ]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
transport.host: 0.0.0.0
  #transport.ping_schedule: 1800s

if I misunderstood you, please correct me.

My template logstash output:

output {
  elasticsearch {
        hosts => ["https://ingest01.my:9200", "https://ingest02.my:9200"]
          index => "index-%{+YYYY.MM}"
          user => "username"
          password => "password"
          ssl => true
          cacert => "/usr/local/share/ca-certificates/elastic-stack-ca-pub-prod.crt"
          ssl_certificate_verification => true
         }
		 }

The data is being transferred, then problems arise with it, I can’t understand what the reason is.

Thanks

6

To check I usually use:

curl -X GET -u elastic:'mypassword' https://master01.my:9200
{
  "name" : "master01.my",
  "cluster_name" : "my-elk",
  "cluster_uuid" : "ACjIGY_AT-6yr0tCdySHqw",
  "version" : {
    "number" : "8.12.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "48a287ab9497e852de30327444b0809e55d46466",
    "build_date" : "2024-02-19T10:04:32.774273190Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

this time it passed, but when there are problems then it gives a 503 error

7

Does anyone have any ideas about this problem? I noticed that the problem is with ingests. This does not happen simultaneously; first, one node may be unavailable (the master cannot be seen), and then after a while the second.

8

Does anyone else have any other ideas?

9

On what type of hardware is the cluster hosted?

What is the size and hardware specification (CPU cores, RAM, heap size and type of storage used) for the different node types in the cluster?

What is the full output of the cluster stats API?

10

Hi. The cluster's virtual machines are hosted on physical servers.
Size cluster elk
4 * database CPU=12 RAM=64 HDD-os=120 HDD-db= 2.5Tb -Xms31g -Xmx31g
3 * master CPU=8 RAM=8 HDD-os=100 -Xms4g -Xmx4g
2 * ingest CPU=8 RAM=32 HDD-os=100 -Xms20g -Xmx20g
2 * logstash CPU=8 RAM=32 HDD-os=300

GET /_cluster/stats

{
  "_nodes": {
    "total": 11,
    "successful": 11,
    "failed": 0
  },
  "cluster_name": "dk-elk-prod",
  "cluster_uuid": "ACjIGY_AT-6yr0tCdySHqw",
  "timestamp": 1724828854058,
  "status": "green",
  "indices": {
    "count": 658,
    "shards": {
      "total": 1495,
      "primaries": 1027,
      "replication": 0.45569620253164556,
      "index": {
        "shards": {
          "min": 1,
          "max": 8,
          "avg": 2.2720364741641337
        },
        "primaries": {
          "min": 1,
          "max": 4,
          "avg": 1.560790273556231
        },
        "replication": {
          "min": 0,
          "max": 3,
          "avg": 0.5531914893617021
        }
      }
    },
    "docs": {
      "count": 12722478124,
      "deleted": 2154271,
      "total_size_in_bytes": 4955891025344
    },
    "store": {
      "size_in_bytes": 5472375473625,
      "total_data_set_size_in_bytes": 5472375473625,
      "reserved_in_bytes": 0
    },
    "fielddata": {
      "memory_size_in_bytes": 135848,
      "evictions": 506,
      "global_ordinals": {
        "build_time_in_millis": 35183
      }
    },
    "query_cache": {
      "memory_size_in_bytes": 66824385,
      "total_count": 10629996,
      "hit_count": 382803,
      "miss_count": 10247193,
      "cache_size": 5321,
      "cache_count": 8548,
      "evictions": 3227
    },
    "completion": {
      "size_in_bytes": 0
    },
    "segments": {
      "count": 49002,
      "memory_in_bytes": 0,
      "terms_memory_in_bytes": 0,
      "stored_fields_memory_in_bytes": 0,
      "term_vectors_memory_in_bytes": 0,
      "norms_memory_in_bytes": 0,
      "points_memory_in_bytes": 0,
      "doc_values_memory_in_bytes": 0,
      "index_writer_memory_in_bytes": 313587476,
      "version_map_memory_in_bytes": 0,
      "fixed_bit_set_memory_in_bytes": 38151688,
      "max_unsafe_auto_id_timestamp": 1724827542242,
      "file_sizes": {}
    },
    "mappings": {
      "total_field_count": 63656,
      "total_deduplicated_field_count": 27916,
      "total_deduplicated_mapping_size_in_bytes": 167093,
      "field_types": [
        {
          "name": "alias",
          "count": 77,
          "index_count": 2,
          "script_count": 0
        },
        {
          "name": "binary",
          "count": 2,
          "index_count": 2,
          "script_count": 0
        },
        {
          "name": "boolean",
          "count": 443,
          "index_count": 87,
          "script_count": 0
        },
        {
          "name": "byte",
          "count": 14,
          "index_count": 14,
          "script_count": 0
        },
        {
          "name": "constant_keyword",
          "count": 7,
          "index_count": 3,
          "script_count": 0
        },
        {
          "name": "date",
          "count": 1686,
          "index_count": 596,
          "script_count": 0
        },
        {
          "name": "date_nanos",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "date_range",
          "count": 16,
          "index_count": 16,
          "script_count": 0
        },
        {
          "name": "double",
          "count": 72,
          "index_count": 19,
          "script_count": 0
        },
        {
          "name": "double_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "flattened",
          "count": 202,
          "index_count": 26,
          "script_count": 0
        },
        {
          "name": "float",
          "count": 850,
          "index_count": 93,
          "script_count": 0
        },
        {
          "name": "float_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "geo_point",
          "count": 68,
          "index_count": 20,
          "script_count": 0
        },
        {
          "name": "geo_shape",
          "count": 5,
          "index_count": 5,
          "script_count": 0
        },
        {
          "name": "half_float",
          "count": 78,
          "index_count": 22,
          "script_count": 0
        },
        {
          "name": "integer",
          "count": 178,
          "index_count": 20,
          "script_count": 0
        },
        {
          "name": "integer_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "ip",
          "count": 89,
          "index_count": 9,
          "script_count": 0
        },
        {
          "name": "ip_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "keyword",
          "count": 22865,
          "index_count": 602,
          "script_count": 0
        },
        {
          "name": "long",
          "count": 5349,
          "index_count": 202,
          "script_count": 0
        },
        {
          "name": "long_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "match_only_text",
          "count": 465,
          "index_count": 15,
          "script_count": 0
        },
        {
          "name": "nested",
          "count": 175,
          "index_count": 36,
          "script_count": 0
        },
        {
          "name": "object",
          "count": 9869,
          "index_count": 589,
          "script_count": 0
        },
        {
          "name": "rank_features",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "scaled_float",
          "count": 34,
          "index_count": 8,
          "script_count": 0
        },
        {
          "name": "shape",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "short",
          "count": 24,
          "index_count": 9,
          "script_count": 0
        },
        {
          "name": "text",
          "count": 20938,
          "index_count": 563,
          "script_count": 0
        },
        {
          "name": "version",
          "count": 32,
          "index_count": 32,
          "script_count": 0
        },
        {
          "name": "wildcard",
          "count": 110,
          "index_count": 5,
          "script_count": 0
        }
      ],
      "runtime_field_types": [
        {
          "name": "double",
          "count": 1,
          "index_count": 1,
          "scriptless_count": 0,
          "shadowed_count": 0,
          "lang": [
            "painless"
          ],
          "lines_max": 1,
          "lines_total": 1,
          "chars_max": 58,
          "chars_total": 58,
          "source_max": 0,
          "source_total": 0,
          "doc_max": 1,
          "doc_total": 1
        },
        {
          "name": "long",
          "count": 2,
          "index_count": 1,
          "scriptless_count": 0,
          "shadowed_count": 0,
          "lang": [
            "painless"
          ],
          "lines_max": 1,
          "lines_total": 2,
          "chars_max": 187,
          "chars_total": 226,
          "source_max": 0,
          "source_total": 0,
          "doc_max": 4,
          "doc_total": 5
        }
      ]
    },
    "analysis": {
      "char_filter_types": [],
      "tokenizer_types": [],
      "filter_types": [],
      "analyzer_types": [],
      "built_in_char_filters": [],
      "built_in_tokenizers": [],
      "built_in_filters": [],
      "built_in_analyzers": [],
      "synonyms": {}
    },
    "versions": [
      {
        "version": "7.10.1",
        "index_count": 5,
        "primary_shard_count": 6,
        "total_primary_bytes": 16957237
      },
      {
        "version": "7.14.0",
        "index_count": 12,
        "primary_shard_count": 14,
        "total_primary_bytes": 500183364
      },
      {
        "version": "7.16.3",
        "index_count": 2,
        "primary_shard_count": 2,
        "total_primary_bytes": 293741580
      },
      {
        "version": "8.3.3",
        "index_count": 32,
        "primary_shard_count": 34,
        "total_primary_bytes": 18047812037
      },
      {
        "version": "8.9.1",
        "index_count": 25,
        "primary_shard_count": 36,
        "total_primary_bytes": 62065484882
      },
      {
        "version": "8.10.2",
        "index_count": 22,
        "primary_shard_count": 36,
        "total_primary_bytes": 61058131682
      },
      {
        "version": "8.11.0-8.11.4",
        "index_count": 57,
        "primary_shard_count": 112,
        "total_primary_bytes": 177413316193
      },
      {
        "version": "8.12.1-8.12.2",
        "index_count": 114,
        "primary_shard_count": 306,
        "total_primary_bytes": 1682537151581
      },
      {
        "version": "8.13.0-8.13.4",
        "index_count": 234,
        "primary_shard_count": 293,
        "total_primary_bytes": 2390088589605
      },
      {
        "version": "8.14.0-8.14.3",
        "index_count": 31,
        "primary_shard_count": 31,
        "total_primary_bytes": 216921260516
      },
      {
        "version": "8.14.4-snapshot[8512000]",
        "index_count": 124,
        "primary_shard_count": 157,
        "total_primary_bytes": 374711120321
      }
    ],
    "search": {
      "total": 1425119,
      "queries": {
        "bool": 1110830,
        "prefix": 9,
        "match": 307196,
        "range": 660019,
        "nested": 24108,
        "wildcard": 10,
        "multi_match": 181,
        "match_phrase": 3582,
        "terms": 628286,
        "constant_score": 608,
        "match_all": 5669,
        "ids": 12,
        "match_phrase_prefix": 450,
        "exists": 381221,
        "term": 1103625,
        "query_string": 306784,
        "simple_query_string": 28480
      },
      "rescorers": {},
      "sections": {
        "highlight": 3221,
        "search_after": 4512,
        "stored_fields": 3697,
        "runtime_mappings": 154273,
        "query": 1423078,
        "script_fields": 3697,
        "pit": 48392,
        "_source": 318140,
        "terminate_after": 524,
        "fields": 5358,
        "collapse": 268245,
        "aggs": 395200
      }
    },
    "dense_vector": {
      "value_count": 0
    },
    "sparse_vector": {
      "value_count": 0
    }
  },
  "nodes": {
    "count": {
      "total": 11,
      "coordinating_only": 2,
      "data": 4,
      "data_cold": 0,
      "data_content": 0,
      "data_frozen": 0,
      "data_hot": 0,
      "data_warm": 0,
      "index": 0,
      "ingest": 2,
      "master": 3,
      "ml": 0,
      "remote_cluster_client": 0,
      "search": 0,
      "transform": 2,
      "voting_only": 0
    },
    "versions": [
      "8.15.0"
    ],
    "os": {
      "available_processors": 78,
      "allocated_processors": 78,
      "names": [
        {
          "name": "Linux",
          "count": 11
        }
      ],
      "pretty_names": [
        {
          "pretty_name": "Ubuntu 22.04.4 LTS",
          "count": 11
        }
      ],
      "architectures": [
        {
          "arch": "amd64",
          "count": 11
        }
      ],
      "mem": {
        "total_in_bytes": 378606067712,
        "adjusted_total_in_bytes": 378606067712,
        "free_in_bytes": 24219238400,
        "used_in_bytes": 354386829312,
        "free_percent": 6,
        "used_percent": 94
      }
    },
    "process": {
      "cpu": {
        "percent": 95
      },
      "open_file_descriptors": {
        "min": 711,
        "max": 4456,
        "avg": 1962
      }
    },
    "jvm": {
      "max_uptime_in_millis": 770373975,
      "versions": [
        {
          "version": "22.0.1",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "vm_version": "22.0.1+8-16",
          "vm_vendor": "Oracle Corporation",
          "bundled_jdk": true,
          "using_bundled_jdk": true,
          "count": 11
        }
      ],
      "mem": {
        "heap_used_in_bytes": 82062098136,
        "heap_max_in_bytes": 197568495616
      },
      "threads": 1155
    },
    "fs": {
      "total_in_bytes": 11386492100608,
      "free_in_bytes": 5575787274240,
      "available_in_bytes": 5062215991296
    },
    "plugins": [],
    "network_types": {
      "transport_types": {
        "security4": 11
      },
      "http_types": {
        "security4": 11
      }
    },
    "discovery_types": {
      "multi-node": 11
    },
    "packaging_types": [
      {
        "flavor": "default",
        "type": "deb",
        "count": 11
      }
    ],
    "ingest": {
      "number_of_pipelines": 62,
      "processor_stats": {
        "append": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "attachment": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "convert": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "csv": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "date": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "date_index_name": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "dot_expander": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "enrich": {
          "count": 14627722,
          "failed": 33659,
          "current": 0,
          "time_in_millis": 3299370695
        },
        "foreach": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "geoip": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "grok": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "gsub": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "inference": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "json": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "pipeline": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "remove": {
          "count": 1339545,
          "failed": 0,
          "current": 0,
          "time_in_millis": 239
        },
        "rename": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "reroute": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "script": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "set": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "set_security_user": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "split": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "trim": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "uri_parts": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "user_agent": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        }
      }
    },
    "indexing_pressure": {
      "memory": {
        "current": {
          "combined_coordinating_and_primary_in_bytes": 0,
          "coordinating_in_bytes": 0,
          "primary_in_bytes": 0,
          "replica_in_bytes": 0,
          "all_in_bytes": 0
        },
        "total": {
          "combined_coordinating_and_primary_in_bytes": 0,
          "coordinating_in_bytes": 0,
          "primary_in_bytes": 0,
          "replica_in_bytes": 0,
          "all_in_bytes": 0,
          "coordinating_rejections": 0,
          "primary_rejections": 0,
          "replica_rejections": 0,
          "primary_document_rejections": 0
        },
        "limit_in_bytes": 0
      }
    }
  },
  "snapshots": {
    "current_counts": {
      "snapshots": 0,
      "shard_snapshots": 0,
      "snapshot_deletions": 0,
      "concurrent_operations": 0,
      "cleanups": 0
    },
    "repositories": {
      "nfs-server": {
        "type": "fs",
        "current_counts": {
          "snapshots": 0,
          "clones": 0,
          "finalizations": 0,
          "deletions": 0,
          "snapshot_deletions": 0,
          "active_deletions": 0,
          "shards": {
            "total": 0,
            "complete": 0,
            "incomplete": 0,
            "states": {
              "INIT": 0,
              "SUCCESS": 0,
              "FAILED": 0,
              "ABORTED": 0,
              "MISSING": 0,
              "WAITING": 0,
              "QUEUED": 0,
              "PAUSED_FOR_NODE_REMOVAL": 0
            }
          }
        },
        "oldest_start_time_millis": 1724828853772
      },
      "my_securuty": {
        "type": "fs",
        "current_counts": {
          "snapshots": 0,
          "clones": 0,
          "finalizations": 0,
          "deletions": 0,
          "snapshot_deletions": 0,
          "active_deletions": 0,
          "shards": {
            "total": 0,
            "complete": 0,
            "incomplete": 0,
            "states": {
              "INIT": 0,
              "SUCCESS": 0,
              "FAILED": 0,
              "ABORTED": 0,
              "MISSING": 0,
              "WAITING": 0,
              "QUEUED": 0,
              "PAUSED_FOR_NODE_REMOVAL": 0
            }
          }
        },
        "oldest_start_time_millis": 1724828853772
      }
    }
  }
}

GET /_health_report/master_is_stable

{
  "cluster_name": "dk-elk-prod",
  "indicators": {
    "master_is_stable": {
      "status": "green",
      "symptom": "The cluster has a stable master node",
      "details": {
        "current_master": {
          "node_id": "J5e2urQZT0enBQgEtqxXSA",
          "name": "v-elk-em01.dkgroup.tek.loc"
        },
        "recent_masters": [
          {
            "node_id": "J5e2urQZT0enBQgEtqxXSA",
            "name": "v-elk-em01.dkgroup.tek.loc"
          }
        ]
      }
    }
  }
}

GET /_cluster/settings?include_defaults=true

{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "balance": {
            "disk_usage": "1.0"
          }
        }
      }
    },
    "xpack": {
      "monitoring": {
        "collection": {
          "enabled": "true"
        }
      }
    }
  },
  "transient": {},
  "defaults": {
    "cluster": {
      "max_voting_config_exclusions": "10",
      "no_master_block": "write",
      "persistent_tasks": {
        "allocation": {
          "enable": "all",
          "recheck_interval": "30s"
        }
      },
      "auto_sharding": {
        "min_write_threads": "2",
        "max_write_threads": "32"
      },
      "remote": {
        "initial_connect_timeout": "30s",
        "node": {
          "attr": ""
        },
        "connections_per_cluster": "3"
      },
      "lifecycle": {
        "default": {
          "rollover": "max_age=auto,max_primary_shard_size=50gb,min_docs=1,max_primary_shard_docs=200000000"
        }
      },
      "routing": {
        "use_adaptive_replica_selection": "true",
        "rebalance": {
          "enable": "all"
        },
        "allocation": {
          "enforce_default_tier_preference": "true",
          "node_concurrent_incoming_recoveries": "2",
          "node_initial_primaries_recoveries": "4",
          "desired_balance": {
            "progress_log_interval": "1m",
            "undesired_allocations": {
              "log_interval": "1h",
              "threshold": "0.1"
            }
          },
          "same_shard": {
            "host": "false"
          },
          "total_shards_per_node": "-1",
          "type": "desired_balance",
          "disk": {
            "threshold_enabled": "true",
            "reroute_interval": "60s",
            "watermark": {
              "flood_stage.frozen.max_headroom": "20GB",
              "flood_stage": "95%",
              "high": "90%",
              "low": "85%",
              "flood_stage.frozen": "95%",
              "flood_stage.max_headroom": "100GB",
              "low.max_headroom": "200GB",
              "enable_for_single_data_node": "true",
              "high.max_headroom": "150GB"
            }
          },
          "awareness": {
            "attributes": []
          },
          "balance": {
            "index": "0.55",
            "threshold": "1.0",
            "shard": "0.45",
            "write_load": "10.0"
          },
          "enable": "all",
          "node_concurrent_outgoing_recoveries": "2",
          "allow_rebalance": "indices_all_active",
          "cluster_concurrent_rebalance": "2",
          "node_concurrent_recoveries": "2"
        }
      },
      "max_shards_per_node.frozen": "3000",
      "deprecation_indexing": {
        "flush_interval": "5s",
        "enabled": "true",
        "x_opaque_id_used": {
          "enabled": "true"
        }
      },
      "info": {
        "update": {
          "interval": "30s",
          "timeout": "15s"
        }
      },
      "auto_shrink_voting_configuration": "true",
      "discovery_configuration_check": {
        "interval": "30000ms"
      },
      "election": {
        "duration": "500ms",
        "initial_timeout": "100ms",
        "max_timeout": "10s",
        "back_off_time": "100ms",
        "strategy": "supports_voting_only"
      },
      "blocks": {
        "read_only_allow_delete": "false",
        "read_only": "false"
      },
      "follower_lag": {
        "timeout": "90000ms"
      },
      "indices": {
        "tombstones": {
          "size": "500"
        },
        "close": {
          "enable": "true"
        }
      },
      "join_validation": {
        "cache_timeout": "60s"
      },
      "nodes": {
        "reconnect_interval": "10s"
      },
      "logsdb": {
        "enabled": "false"
      },
      "service": {
        "slow_master_task_logging_threshold": "10s",
        "slow_task_logging_threshold": "30s",
        "master_service_starvation_logging_threshold": "5m",
        "slow_task_thread_dump_timeout": "30s"
      },
      "publish": {
        "timeout": "30000ms",
        "info_timeout": "10000ms"
      },
      "name": "dk-elk-prod",
      "fault_detection": {
        "leader_check": {
          "interval": "1000ms",
          "timeout": "10000ms",
          "retry_count": "3"
        },
        "follower_check": {
          "interval": "1000ms",
          "timeout": "10000ms",
          "retry_count": "3"
        }
      },
      "max_shards_per_node": "1000",
      "initial_master_nodes": [],
      "snapshot": {
        "info": {
          "max_concurrent_fetches": "5"
        }
      }
    },
    "stack": {
      "templates": {
        "enabled": "true"
      }
    },
    "time_series": {
      "poll_interval": "5m"
    },
    "logger": {
      "level": "INFO"
    },
    "esql_worker": {
      "queue_size": "1000",
      "size": "7"
    },
    "health_node": {
      "transport_action_timeout": "5s"
    },
    "ingest": {
      "user_agent": {
        "cache_size": "1000"
      },
      "geoip": {
        "cache_size": "1000",
        "downloader": {
          "endpoint": "https://geoip.elastic.co/v1/database",
          "poll": {
            "interval": "3d"
          },
          "eager": {
            "download": "false"
          },
          "enabled": "true"
        }
      },
      "grok": {
        "watchdog": {
          "max_execution_time": "1s",
          "interval": "1s"
        }
      }
    },
    "path": {
      "data": [
        "/var/lib/elasticsearch"
      ],
      "logs": "/var/log/elasticsearch",
      "shared_data": "",
      "home": "/usr/share/elasticsearch",
      "repo": [
        "/mnt/nfs-share"
      ]
    },
    "ccr": {
      "wait_for_metadata_timeout": "60s",
      "indices": {
        "recovery": {
          "recovery_activity_timeout": "60s",
          "chunk_size": "1mb",
          "internal_action_timeout": "60s",
          "max_bytes_per_sec": "40mb",
          "max_concurrent_file_chunks": "5"
        }
      },
      "auto_follow": {
        "wait_for_metadata_timeout": "60s"
      }
    },
    "remote_cluster": {
      "tcp": {
        "reuse_address": "true",
        "keep_count": "-1",
        "keep_interval": "-1",
        "no_delay": "true",
        "keep_alive": "true",
        "keep_idle": "-1",
        "send_buffer_size": "-1b"
      },
      "bind_host": [
        "0.0.0.0"
      ],
      "port": "9443",
      "host": [
        "0.0.0.0"
      ],
      "max_request_header_size": "64kb",
      "publish_port": "-1",
      "publish_host": [
        "0.0.0.0"
      ]
    },
    "repositories": {
      "fs": {
        "chunk_size": "9223372036854775807b",
        "location": ""
      },
      "url": {
        "supported_protocols": [
          "http",
          "https",
          "ftp",
          "file",
          "jar"
        ],
        "allowed_urls": [],
        "url": "http:"
      }
    },
    "action": {
      "auto_create_index": "true",
      "search": {
        "pre_filter_shard_size": {
          "default": "128"
        },
        "shard_count": {
          "limit": "9223372036854775807"
        }
      },
      "destructive_requires_name": "true"
    },
    "enrich": {
      "max_force_merge_attempts": "3",
      "cleanup_period": "15m",
      "fetch_size": "10000",
      "cache_size": "1000",
      "coordinator_proxy": {
        "max_concurrent_requests": "8",
        "max_lookups_per_request": "128",
        "queue_capacity": "1024"
      },
      "max_concurrent_policy_executions": "50"
    },
    "repository_s3": {
      "compare_and_exchange": {
        "time_to_live": "15s",
        "anti_contention_delay": "1s"
      }
    },
    "cache": {
      "recycler": {
        "page": {
          "limit": {
            "heap": "10%"
          },
          "type": "CONCURRENT",
          "weight": {
            "longs": "1.0",
            "ints": "1.0",
            "bytes": "1.0",
            "objects": "0.1"
          }
        }
      }
    },
    "tracing": {
      "apm": {
        "sanitize_field_names": [
          "password",
          "passwd",
          "pwd",
          "secret",
          "*key",
          "*token*",
          "*session*",
          "*credit*",
          "*card*",
          "*auth*",
          "*principal*",
          "set-cookie"
        ],
        "enabled": "false",
        "names": {
          "include": [],
          "exclude": []
        }
      }
    },
    "reindex": {
      "remote": {
        "whitelist": []
      }
    },
    "resource": {
      "reload": {
        "enabled": "true",
        "interval": {
          "low": "60s",
          "high": "5s",
          "medium": "30s"
        }
      }
    },
    "thread_pool": {
      "force_merge": {
        "queue_size": "-1",
        "size": "1"
      },
      "search_coordination": {
        "queue_size": "1000",
        "size": "2"
      },
      "snapshot_meta": {
        "core": "1",
        "max": "12",
        "keep_alive": "30s"
      },
      "fetch_shard_started": {
        "core": "1",
        "max": "8",
        "keep_alive": "5m"
      },
      "estimated_time_interval.warn_threshold": "5s",
      "scheduler": {
        "warn_threshold": "5s"
      },
      "cluster_coordination": {
        "queue_size": "-1",
        "size": "1"
      },
      "search": {
        "queue_size": "1000",
        "size": "7"
      },
      "fetch_shard_store": {
        "core": "1",
        "max": "8",
        "keep_alive": "5m"
      },
      "flush": {
        "core": "1",
        "max": "2",
        "keep_alive": "5m"
      },
      "get": {
        "queue_size": "1000",
        "size": "7"
      },
      "system_read": {
        "queue_size": "2000",
        "size": "2"
      },
      "system_critical_read": {
        "queue_size": "2000",
        "size": "2"
      },
      "estimated_time_interval": "200ms",
      "write": {
        "queue_size": "10000",
        "size": "4"
      },
      "search_worker": {
        "queue_size": "-1",
        "size": "7"
      },
      "system_critical_write": {
        "queue_size": "1500",
        "size": "2"
      },
      "refresh": {
        "core": "1",
        "max": "2",
        "keep_alive": "5m"
      },
      "repository_azure": {
        "core": "0",
        "max": "5",
        "keep_alive": "30s"
      },
      "system_write": {
        "queue_size": "1000",
        "size": "2"
      },
      "generic": {
        "core": "4",
        "max": "128",
        "keep_alive": "30s"
      },
      "warmer": {
        "core": "1",
        "max": "2",
        "keep_alive": "5m"
      },
      "auto_complete": {
        "queue_size": "100",
        "size": "1"
      },
      "azure_event_loop": {
        "core": "0",
        "max": "1",
        "keep_alive": "30s"
      },
      "profiling": {
        "core": "0",
        "max": "1",
        "keep_alive": "30m"
      },
      "management": {
        "core": "1",
        "max": "4",
        "keep_alive": "5m"
      },
      "analyze": {
        "queue_size": "16",
        "size": "1"
      },
      "snapshot": {
        "core": "1",
        "max": "10",
        "keep_alive": "5m"
      },
      "search_throttled": {
        "queue_size": "100",
        "size": "1"
      }
    },
    "index": {
      "codec": "default",
      "recovery": {
        "type": ""
      },
      "store": {
        "type": "",
        "fs": {
          "fs_lock": "native"
        },
        "preload": [],
        "snapshot": {
          "uncached_chunk_size": "-1b",
          "cache": {
            "excluded_file_types": []
          }
        }
      }
    },
    "runtime_fields": {
      "grok": {
        "watchdog": {
          "max_execution_time": "1s",
          "interval": "1s"
        }
      }
    },
    "cluster_state": {
      "document_page_size": "1mb"
    },
    "deprecation": {
      "skip_deprecated_settings": []
    },
    "script": {
      "allowed_contexts": [],
      "max_compilations_rate": "150/5m",
      "cache": {
        "max_size": "3000",
        "expire": "0ms"
      },
      "painless": {
        "regex": {
          "enabled": "limited",
          "limit-factor": "6"
        }
      },
      "max_size_in_bytes": "65535",
      "allowed_types": [],
      "disable_max_compilations_rate": "false"
    },
    "indexing_pressure": {
      "memory": {
        "limit": "10%"
      }
    },
    "node": {
      "bandwidth": {
        "recovery": {
          "disk": {
            "write": "-1",
            "read": "-1"
          },
          "factor": {
            "write": "0.4",
            "read": "0.4"
          },
          "operator": {
            "factor.read": "0.4",
            "factor.write": "0.4",
            "factor": "0.4",
            "factor.max_overcommit": "100.0"
          },
          "network": "-1"
        }
      },
      "enable_lucene_segment_infos_trace": "false",
      "roles": [],
      "_internal": {
        "default_refresh_interval": "1s"
      },
      "name": "v-elk-ekb01.dkgroup.tek.loc",
      "external_id": "",
      "id": {
        "seed": "0"
      },
      "processors": "4.0",
      "store": {
        "allow_mmap": "true"
      },
      "attr": {
        "transform": {
          "config_version": "10.0.0"
        },
        "xpack": {
          "installed": "true"
        },
        "ml": {
          "config_version": "12.0.0"
        }
      },
      "portsfile": "false"
    },
    "slm": {
      "health": {
        "failed_snapshot_warn_threshold": "5"
      },
      "minimum_interval": "15m",
      "retention_schedule": "0 30 1 * * ?",
      "retention_duration": "1h",
      "history_index_enabled": "true"
    },
    "http": {
      "cors": {
        "max-age": "1728000",
        "allow-origin": "",
        "allow-headers": "X-Requested-With,Content-Type,Content-Length,Authorization,Accept,User-Agent,X-Elastic-Client-Meta",
        "allow-credentials": "false",
        "allow-methods": "OPTIONS,HEAD,GET,POST,PUT,DELETE",
        "enabled": "false"
      },
      "max_chunk_size": "8kb",
      "compression_level": "3",
      "max_initial_line_length": "4kb",
      "shutdown_grace_period": "0ms",
      "type": "security4",
      "pipelining": {
        "max_events": "10000"
      },
      "shutdown_poll_period": "5m",
      "type.default": "netty4",
      "host": [
        "0.0.0.0"
      ],
      "publish_port": "-1",
      "read_timeout": "0ms",
      "max_content_length": "100mb",
      "netty": {
        "receive_predictor_size": "64kb",
        "max_composite_buffer_components": "69905",
        "worker_count": "0"
      },
      "tcp": {
        "reuse_address": "true",
        "keep_count": "-1",
        "keep_interval": "-1",
        "no_delay": "true",
        "keep_alive": "true",
        "receive_buffer_size": "-1b",
        "keep_idle": "-1",
        "send_buffer_size": "-1b"
      },
      "bind_host": [
        "0.0.0.0"
      ],
      "client_stats": {
        "enabled": "true",
        "closed_channels": {
          "max_age": "5m",
          "max_count": "10000"
        }
      },
      "reset_cookies": "false",
      "max_warning_header_count": "-1",
      "tracer": {
        "include": [],
        "exclude": []
      },
      "max_warning_header_size": "-1b",
      "detailed_errors": {
        "enabled": "true"
      },
      "port": "9200",
      "max_header_size": "16kb",
      "compression": "false",
      "publish_host": [
        "0.0.0.0"
      ]
    },
    "telemetry": {
      "tracing": {
        "sanitize_field_names": [
          "password",
          "passwd",
          "pwd",
          "secret",
          "*key",
          "*token*",
          "*session*",
          "*credit*",
          "*card*",
          "*auth*",
          "*principal*",
          "set-cookie"
        ],
        "enabled": "false",
        "names": {
          "include": [],
          "exclude": []
        }
      },
      "metrics": {
        "enabled": "false"
      }
    },
    "snapshot": {
      "refresh_repo_uuid_on_restore": "true",
      "max_concurrent_operations": "1000"
    },
    "readiness": {
      "port": "-1"
    },
    "bootstrap": {
      "memory_lock": "true",
      "ctrlhandler": "true"
    },
    "network": {
      "tcp": {
        "reuse_address": "true",
        "keep_count": "-1",
        "keep_interval": "-1",
        "no_delay": "true",
        "keep_alive": "true",
        "receive_buffer_size": "-1b",
        "keep_idle": "-1",
        "send_buffer_size": "-1b"
      },
      "bind_host": [
        "0"
      ],
      "server": "true",
      "breaker": {
        "inflight_requests": {
          "limit": "100%",
          "overhead": "2.0"
        }
      },
      "host": [
        "0"
      ],
      "thread": {
        "watchdog": {
          "quiet_time": "10m",
          "interval": "5s"
        }
      },
      "publish_host": [
        "0"
      ]
    },
    "searchable_snapshots": {
      "blob_cache": {
        "periodic_cleanup": {
          "interval": "1h",
          "batch_size": "100",
          "pit_keep_alive": "10m",
          "retention_period": "1h"
        }
      }
    },
    "search": {
      "default_search_timeout": "-1",
      "query_phase_parallel_collection_enabled": "true",
      "max_open_scroll_context": "500",
      "max_buckets": "65536",
      "max_async_search_response_size": "10mb",
      "worker_threads_enabled": "true",
      "keep_alive_interval": "1m",
      "max_keep_alive": "24h",
      "highlight": {
        "term_vector_multi_value": "true"
      },
      "default_allow_partial_results": "true",
      "low_level_cancellation": "true",
      "allow_expensive_queries": "true",
      "check_ccs_compatibility": "false",
      "default_keep_alive": "5m",
      "aggs": {
        "only_allowed_metric_scripts": "false",
        "allowed_inline_metric_scripts": [],
        "rewrite_to_filter_by_filter": "true",
        "tdigest_execution_hint": "DEFAULT",
        "allowed_stored_metric_scripts": []
      }
    },
    "security": {
      "manager": {
        "filter_bad_defaults": "true"
      }
    },
    "client": {
      "type": "node"
    },
    "xpack": {
      "watcher": {
        "execution": {
          "scroll": {
            "size": "0",
            "timeout": ""
          },
          "default_throttle_period": "5s"
        },
        "internal": {
          "ops": {
            "bulk": {
              "default_timeout": ""
            },
            "index": {
              "default_timeout": ""
            },
            "search": {
              "default_timeout": ""
            }
          }
        },
        "thread_pool": {
          "queue_size": "1000",
          "size": "1"
        },
        "index": {
          "rest": {
            "direct_access": ""
          }
        },
        "use_ilm_index_management": "true",
        "trigger": {
          "schedule": {
            "ticker": {
              "tick_interval": "500ms"
            }
          }
        },
        "enabled": "true",
        "input": {
          "search": {
            "default_timeout": ""
          }
        },
        "encrypt_sensitive_data": "false",
        "transform": {
          "search": {
            "default_timeout": ""
          }
        },
        "stop": {
          "timeout": "30s"
        },
        "watch": {
          "scroll": {
            "size": "0"
          }
        },
        "bulk": {
          "concurrent_requests": "0",
          "flush_interval": "1s",
          "size": "1mb",
          "actions": "1"
        },
        "actions": {
          "bulk": {
            "default_timeout": ""
          },
          "index": {
            "default_timeout": ""
          }
        }
      },
      "eql": {
        "enabled": "true"
      },
      "inference": {
        "utility_thread_pool": {
          "core": "0",
          "max": "10",
          "keep_alive": "10m"
        },
        "http": {
          "max_total_connections": "50",
          "request_executor": {
            "task_poll_frequency": "50ms",
            "rate_limit_group_cleanup_interval": "1d",
            "rate_limit_group_stale_duration": "10d",
            "queue_capacity": "2000"
          },
          "max_route_connections": "20",
          "max_response_size": "50mb",
          "connection_eviction_interval": "1m",
          "connection_eviction_max_idle_time": "1m",
          "retry": {
            "debug_frequency_amount": "5m",
            "initial_delay": "1s",
            "debug_frequency_mode": "OFF",
            "max_delay_bound": "5s",
            "timeout": "30s"
          }
        },
        "logging": {
          "reset_interval": "1d",
          "wait_duration": "1h"
        },
        "skip_validate_and_start": "false",
        "truncator": {
          "reduction_percentage": "0.5"
        }
      },
      "ent_search": {
        "enabled": "true"
      },
      "monitoring": {
        "migration": {
          "decommission_alerts": "false"
        },
        "collection": {
          "cluster": {
            "stats": {
              "timeout": "10s"
            }
          },
          "node": {
            "stats": {
              "timeout": "10s"
            }
          },
          "indices": [],
          "ccr": {
            "stats": {
              "timeout": "10s"
            }
          },
          "enrich": {
            "stats": {
              "timeout": "10s"
            }
          },
          "index": {
            "stats": {
              "timeout": "10s"
            },
            "recovery": {
              "active_only": "false",
              "timeout": "10s"
            }
          },
          "interval": "10s",
          "ml": {
            "job": {
              "stats": {
                "timeout": "10s"
              }
            }
          }
        },
        "history": {
          "duration": "168h"
        },
        "elasticsearch": {
          "collection": {
            "enabled": "true"
          }
        },
        "templates": {
          "enabled": "true"
        }
      },
      "graph": {
        "enabled": "true"
      },
      "searchable": {
        "snapshot": {
          "allocate_on_rolling_restart": "false",
          "cache": {
            "range_size": "32mb",
            "sync": {
              "max_files": "10000",
              "interval": "60s",
              "shutdown_timeout": "10s"
            },
            "recovery_range_size": "128kb"
          },
          "shared_cache": {
            "recovery_range_size": "128kb",
            "region_size": "16mb",
            "size": "0",
            "min_time_delta": "60s",
            "decay": {
              "interval": "60s"
            },
            "count_reads": "true",
            "size.max_headroom": "-1",
            "mmap": "false",
            "range_size": "16mb",
            "max_freq": "100"
          }
        }
      },
      "rollup": {
        "task_thread_pool": {
          "queue_size": "-1",
          "size": "1"
        }
      },
      "searchable_snapshots": {
        "cache_fetch_async_thread_pool": {
          "core": "0",
          "max": "12",
          "keep_alive": "30s"
        },
        "cache_prewarming_thread_pool": {
          "core": "0",
          "max": "16",
          "keep_alive": "30s"
        }
      },
      "downsample": {
        "thread_pool": {
          "queue_size": "256",
          "size": "1"
        }
      },
      "license": {
        "upload": {
          "types": [
            "standard",
            "gold",
            "platinum",
            "enterprise",
            "trial"
          ]
        },
        "self_generated": {
          "type": "basic"
        }
      },
      "notification": {
        "pagerduty": {
          "default_account": ""
        },
        "webhook": {
          "additional_token_enabled": "false"
        },
        "email": {
          "account": {
            "domain_allowlist": [
              "*"
            ]
          },
          "default_account": "",
          "html": {
            "sanitization": {
              "allow": [
                "body",
                "head",
                "_tables",
                "_links",
                "_blocks",
                "_formatting",
                "img:embedded"
              ],
              "disallow": [],
              "enabled": "true"
            }
          }
        },
        "reporting": {
          "retries": "40",
          "warning": {
            "enabled": "true"
          },
          "interval": "15s"
        },
        "jira": {
          "default_account": ""
        },
        "slack": {
          "default_account": ""
        }
      },
      "security": {
        "operator_privileges": {
          "enabled": "false"
        },
        "dls_fls": {
          "enabled": "true"
        },
        "dls": {
          "bitset": {
            "cache": {
              "size": "10%",
              "ttl": "2h"
            }
          }
        },
        "transport": {
          "filter": {
            "allow": [],
            "deny": [],
            "enabled": "true"
          },
          "ssl": {
            "enabled": "true"
          }
        },
        "remote_cluster_server": {
          "ssl": {
            "enabled": "true"
          }
        },
        "ssl": {
          "diagnose": {
            "trust": "true"
          }
        },
        "enabled": "true",
        "crypto": {
          "thread_pool": {
            "queue_size": "1000",
            "size": "2"
          }
        },
        "enrollment": {
          "enabled": "false"
        },
        "filter": {
          "always_allow_bound_address": "true"
        },
        "encryption": {
          "algorithm": "AES/CTR/NoPadding"
        },
        "remote_cluster_client": {
          "ssl": {
            "enabled": "true"
          }
        },
        "remote_cluster": {
          "filter": {
            "allow": [],
            "deny": []
          }
        },
        "audit": {
          "enabled": "false",
          "logfile": {
            "emit_cluster_name": "false",
            "emit_node_id": "true",
            "emit_node_name": "false",
            "emit_node_host_address": "false",
            "emit_cluster_uuid": "true",
            "emit_node_host_name": "false",
            "events": {
              "emit_request_body": "false",
              "include": [
                "ACCESS_DENIED",
                "ACCESS_GRANTED",
                "ANONYMOUS_ACCESS_DENIED",
                "AUTHENTICATION_FAILED",
                "CONNECTION_DENIED",
                "TAMPERED_REQUEST",
                "RUN_AS_DENIED",
                "RUN_AS_GRANTED",
                "SECURITY_CONFIG_CHANGE"
              ],
              "exclude": []
            }
          }
        },
        "authc": {
          "password_hashing": {
            "algorithm": "BCRYPT"
          },
          "success_cache": {
            "size": "10000",
            "enabled": "true",
            "expire_after_access": "1h"
          },
          "api_key": {
            "doc_cache": {
              "ttl": "5m"
            },
            "cache": {
              "hash_algo": "ssha256",
              "max_keys": "25000",
              "ttl": "24h"
            },
            "delete": {
              "interval": "24h",
              "retention_period": "7d",
              "timeout": "-1"
            },
            "enabled": "true",
            "hashing": {
              "algorithm": "PBKDF2"
            }
          },
          "anonymous": {
            "authz_exception": "true",
            "roles": [],
            "username": "_anonymous"
          },
          "run_as": {
            "enabled": "true"
          },
          "reserved_realm": {
            "enabled": "true"
          },
          "service_token": {
            "cache": {
              "hash_algo": "ssha256",
              "max_tokens": "100000",
              "ttl": "20m"
            }
          },
          "token": {
            "delete": {
              "interval": "30m",
              "timeout": "-1"
            },
            "enabled": "false",
            "thread_pool": {
              "queue_size": "1000",
              "size": "1"
            },
            "timeout": "20m"
          }
        },
        "autoconfiguration": {
          "enabled": "true"
        },
        "fips_mode": {
          "enabled": "false",
          "required_providers": []
        },
        "encryption_key": {
          "length": "128",
          "algorithm": "AES"
        },
        "http": {
          "filter": {
            "allow": [],
            "deny": [],
            "enabled": "true"
          },
          "ssl": {
            "enabled": "true"
          }
        },
        "automata": {
          "max_determinized_states": "100000",
          "cache": {
            "size": "10000",
            "ttl": "48h",
            "enabled": "true"
          }
        },
        "user": null,
        "authz": {
          "timer": {
            "indices": {
              "enabled": "false",
              "threshold": {
                "warn": "200ms",
                "debug": "20ms",
                "info": "100ms"
              }
            }
          },
          "store": {
            "privileges": {
              "cache": {
                "ttl": "24h",
                "max_size": "10000"
              }
            },
            "roles": {
              "has_privileges": {
                "cache": {
                  "max_size": "1000"
                }
              },
              "cache": {
                "max_size": "10000"
              },
              "negative_lookup_cache": {
                "max_size": "10000"
              },
              "field_permissions": {
                "cache": {
                  "max_size_in_bytes": "104857600"
                }
              }
            }
          }
        }
      },
      "transform": {
        "num_transform_failure_retries": "10",
        "transform_scheduler_frequency": "1s"
      },
      "ccr": {
        "enabled": "true",
        "ccr_thread_pool": {
          "queue_size": "100",
          "size": "32"
        }
      },
      "idp": {
        "privileges": {
          "application": "",
          "cache": {
            "size": "100",
            "ttl": "90m"
          }
        },
        "metadata": {
          "signing": {
            "keystore": {
              "alias": ""
            }
          }
        },
        "slo_endpoint": {
          "post": "https:",
          "redirect": "https:"
        },
        "defaults": {
          "nameid_format": "urn:oasis:names:tc:SAML:2.0:nameid-format:transient",
          "authn_expiry": "5m"
        },
        "allowed_nameid_formats": [
          "urn:oasis:names:tc:SAML:2.0:nameid-format:transient"
        ],
        "contact": {
          "given_name": "",
          "email": "",
          "surname": ""
        },
        "organization": {
          "display_name": "",
          "name": "",
          "url": "http:"
        },
        "sso_endpoint": {
          "post": "https:",
          "redirect": "https:"
        },
        "entity_id": "",
        "signing": {
          "keystore": {
            "alias": ""
          }
        },
        "sp": {
          "cache": {
            "size": "1000",
            "ttl": "60m"
          },
          "wildcard": {
            "path": "wildcard_services.json"
          }
        },
        "enabled": "false"
      },
      "profiling": {
        "check_outdated_indices": "true",
        "enabled": "true",
        "query": {
          "stacktrace": {
            "max_slices": "16"
          },
          "details": {
            "max_slices": "16"
          },
          "realtime": "true"
        },
        "templates": {
          "enabled": "false"
        }
      },
      "http": {
        "tcp": {
          "keep_alive": "true"
        },
        "default_connection_timeout": "10s",
        "proxy": {
          "host": "",
          "scheme": "",
          "port": "0"
        },
        "connection_pool_ttl": "-1",
        "max_response_size": "10mb",
        "whitelist": [
          "*"
        ],
        "default_read_timeout": "10s"
      },
      "autoscaling": {
        "memory": {
          "monitor": {
            "timeout": "15s"
          }
        }
      },
      "apm_data": {
        "registry": {
          "enabled": "true"
        },
        "enabled": "true"
      },
      "applications": {
        "behavioral_analytics": {
          "ingest": {
            "bulk_processor": {
              "max_events_per_bulk": "500",
              "flush_delay": "10s",
              "max_bytes_in_flight": "5%",
              "max_number_of_retries": "1"
            }
          }
        },
        "rules": {
          "max_rules_per_ruleset": "100"
        }
      },
      "ml": {
        "dummy_entity_processors": "0",
        "utility_thread_pool": {
          "core": "1",
          "max": "2048",
          "keep_alive": "10m"
        },
        "enable_config_migration": "true",
        "delayed_data_check_freq": "15m",
        "min_disk_space_off_heap": "5gb",
        "model_repository": "https://ml-models.elastic.co",
        "use_auto_machine_memory_percent": "false",
        "inference_model": {
          "cache_size": "40%",
          "time_to_live": "5m"
        },
        "node_concurrent_job_allocations": "2",
        "enabled": "true",
        "max_ml_node_size": "0b",
        "dummy_entity_memory": "0b",
        "datafeed_thread_pool": {
          "core": "1",
          "max": "512",
          "keep_alive": "1m"
        },
        "native_inference_comms_thread_pool": {
          "core": "3",
          "max": "312",
          "keep_alive": "1m"
        },
        "process_connect_timeout": "10s",
        "job_comms_thread_pool": {
          "core": "4",
          "max": "2048",
          "keep_alive": "1m"
        },
        "max_anomaly_records": "500",
        "max_open_jobs": "512",
        "allocated_processors_scale": "1",
        "nightly_maintenance_requests_per_second": "-1.0",
        "max_model_memory_limit": "0b",
        "max_lazy_ml_nodes": "0",
        "max_machine_memory_percent": "30",
        "persist_results_max_retries": "20",
        "autodetect_process": "true",
        "max_inference_processors": "50"
      }
    },
    "rest": {
      "action": {
        "multi": {
          "allow_explicit_index": "true"
        }
      }
    },
    "async_search": {
      "index_cleanup_interval": "1h"
    },
    "esql": {
      "query": {
        "result_truncation_default_size": "1000",
        "result_truncation_max_size": "10000"
      }
    },
    "health": {
      "periodic_logger": {
        "output_mode": [
          "logs",
          "metrics"
        ],
        "poll_interval": "60s",
        "enabled": "false"
      },
      "node": {
        "enabled": "true"
      },
      "master_history": {
        "has_master_lookup_timeframe": "30s",
        "identity_changes_threshold": "4",
        "no_master_transitions_threshold": "4"
      },
      "ilm": {
        "max_time_on_action": "1d",
        "max_time_on_step": "1d",
        "max_retries_per_step": "100"
      },
      "reporting": {
        "local": {
          "monitor": {
            "interval": "30s"
          }
        }
      }
    },
    "monitor": {
      "jvm": {
11

This message is coming from the operating system and indicates that something outside of ES forcefully closed the connection between the nodes. Typically this "something" is under the control of your network infrastructure folks, so I would ask them for help. See these troubleshooting docs for more information, particularly the bit at the end:

If these logs do not show enough information to diagnose the problem, obtain a packet capture simultaneously from the nodes at both ends of an unstable connection and analyse it alongside the Elasticsearch logs from those nodes to determine if traffic between the nodes is being disrupted by another device on the network.