Hi Team!
Recently I began to notice unstable operation of Logstash, I started looking at the logs (I have two Logstash nodes). Here is an example of logs:
Oct 27 07:01:17 v-elk-lst01.my logstash[42023]: [2023-10-27T07:01:17,652][INFO ][logstash.outputs.elasticsearch][azurevpn-logs-pipe] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
Oct 27 07:01:17 v-elk-lst01.my logstash[42023]: [2023-10-27T07:01:17,652][WARN ][logstash.outputs.elasticsearch][azurevpn-logs-pipe] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://logstash_internal:xxxxxx@v-elk-eig02.my:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://v-elk-eig02.my:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
Oct 27 07:01:17 v-elk-lst01.my logstash[42023]: [2023-10-27T07:01:17,681][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Failed to perform request {:message=>"Connection pool shut down", :exception=>Manticore::ClientStoppedException, :cause=>#<Java::JavaLang::IllegalStateException: Connection pool shut down>}
Oct 27 07:01:17 v-elk-lst01.my logstash[42023]: [2023-10-27T07:01:17,681][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://remote_monitoring_user1:xxxxxx@v-elk-eig01.my:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://v-elk-eig01.my:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
Then I started checking the logs on the following Ingest server (v-elk-eig01):
[2023-10-27T12:48:53,281][INFO ][o.e.t.OutboundHandler ] [v-elk-eig01.my] send message failed [channel: Netty4TcpChannel{localAddress=/v-elk-eig01.my:54042, remoteAddress=v-elk-ed01.my:9301, profile=default}]: Connection timed out
[2023-10-27T12:48:53,281][WARN ][o.e.t.OutboundHandler ] [v-elk-eig01.my] sending transport message [Request{indices:data/write/bulk[s]}{30098725}{false}{true}{false}] of size [1533] on [Netty4TcpChannel{localAddress=/v-elk-eig01.my:54042, remoteAddress=v-elk-ed01.my:9301, profile=default}] took [731987ms] which is above the warn threshold of [5000ms] with success [false]
v-elk-eig02 logs:
[2023-10-27T12:04:05,041][WARN ][o.e.c.c.ClusterFormationFailureHelper] [v-elk-eig02.my] master not discovered yet: have discovered [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]; discovery will continue using [10.1.5.210:9301, 10.1.5.211:9301, 10.1.5.212:9301] from hosts providers and [{v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] from last-known cluster state; node term 45, last-accepted version 449311 in term 45; joining [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] in term [45] has status [waiting for response] after [14.9m/894401ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/discovery-troubleshooting.html
[2023-10-27T12:04:07,346][WARN ][r.suppressed ] [v-elk-eig02.my] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:174) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:60) ~[?:?]
at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:35) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:455) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1015) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:981) ~[?:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:469) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:185) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:459) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:292) ~[?:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:146) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:261) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:199) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:379) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$prepareRequest$0(RestMonitoringBulkAction.java:101) ~[?:?]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:96) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:87) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:82) ~[?:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:415) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:544) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:317) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:453) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:549) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:426) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120) ~[?:?]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[?:?]
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
[2023-10-27T12:04:08,602][WARN ][o.e.x.m.MonitoringService] [v-elk-eig02.my] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:110) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$1(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:171) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.retry(TransportBulkAction.java:657) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:644) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:513) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
... 33 more
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:640) ~[elasticsearch-8.10.2.jar:?]
... 10 more
[2023-10-27T12:04:09,245][WARN ][r.suppressed ] [v-elk-eig02.my] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:640) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:513) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
[2023-10-27T12:04:15,041][WARN ][o.e.c.c.ClusterFormationFailureHelper] [v-elk-eig02.my] master not discovered yet: have discovered [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]; discovery will continue using [10.1.5.210:9301, 10.1.5.211:9301, 10.1.5.212:9301] from hosts providers and [{v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] from last-known cluster state; node term 45, last-accepted version 449311 in term 45; joining [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] in term [45] has status [waiting for response] after [15m/904405ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/discovery-troubleshooting.html
[2023-10-27T12:04:16,763][WARN ][r.suppressed ] [v-elk-eig02.my] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:640) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:513) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
[2023-10-27T12:04:17,531][INFO ][o.e.t.TcpTransport ] [v-elk-eig02.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.6.14:49642, remoteAddress=10.1.5.212/10.1.5.212:9301, profile=default}], disconnecting from relevant node: Connection timed out
[2023-10-27T12:04:17,531][INFO ][o.e.t.ClusterConnectionManager] [v-elk-eig02.my] transport connection to [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] closed by remote
[2023-10-27T12:04:17,531][INFO ][o.e.c.c.JoinHelper ] [v-elk-eig02.my] failed to join {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0} with JoinRequest{sourceNode={v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, transportVersion=8500061, minimumTerm=45, optionalJoin=Optional[Join{term=45, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, targetNode={v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}}]}
org.elasticsearch.transport.NodeDisconnectedException: [v-elk-em03.my][10.1.5.212:9301][internal:cluster/coordination/join] disconnected
[2023-10-27T12:04:18,269][INFO ][o.e.c.s.ClusterApplierService] [v-elk-eig02.my] master node changed {previous [], current [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]}, term: 45, version: 449312, reason: ApplyCommitRequest{term=45, version=449312, sourceNode={v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}}
next step Master logs v-elk-em03:
[2023-10-27T10:00:16,238][INFO ][o.e.c.m.MetadataMappingService] [v-elk-em03.my] [my-ngfw-logs-2023.10.27/0En_oXU7Rrij4nY_REyT0g] update_mapping [_doc]
[2023-10-27T10:23:20,459][INFO ][o.e.c.c.NodeJoinExecutor ] [v-elk-em03.my] node-join: [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}] with reason [rejoining]
[2023-10-27T10:28:24,766][INFO ][o.e.t.TcpTransport ] [v-elk-em03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.212:9301, remoteAddress=/10.1.6.14:54406, profile=default}], disconnecting from relevant node: Connection reset
[2023-10-27T10:42:00,611][INFO ][o.e.c.m.MetadataMappingService] [v-elk-em03.my] [my-ngfw-logs-2023.10.27/0En_oXU7Rrij4nY_REyT0g] update_mapping [_doc]
[2023-10-27T11:36:07,447][INFO ][o.e.c.m.MetadataMappingService] [v-elk-em03.my] [winlogbeat-srv-sdc-2023.10.27/p1Hi2Sx6SamnWsJrFCBM4A] update_mapping [_doc]
[2023-10-27T11:36:54,642][INFO ][o.e.c.c.NodeJoinExecutor ] [v-elk-em03.my] node-join: [{v-elk-eig01.my}{pZK4-zUxTzGbfKIcLFVkwg}{BOjq36m9Tsm98rZo4Huxpw}{v-elk-eig01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.10.2}{7000099-8100299}] with reason [rejoining]
[2023-10-27T11:41:52,062][INFO ][o.e.t.TcpTransport ] [v-elk-em03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.212:9301, remoteAddress=/10.1.6.13:35226, profile=default}], disconnecting from relevant node: Connection reset
2023-10-27T12:04:18,283][INFO ][o.e.c.c.NodeJoinExecutor ] [v-elk-em03.my] node-join: [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}] with reason [rejoining]
[2023-10-27T12:09:26,847][INFO ][o.e.t.TcpTransport ] [v-elk-em03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.212:9301, remoteAddress=/10.1.6.14:49642, profile=default}], disconnecting from relevant node: Connection reset
next step v-elk-em01:
[2023-10-27T02:20:31,153][INFO ][o.e.t.TcpTransport ] [v-elk-em01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.210:9301, remoteAddress=/10.1.6.13:49494, profile=default}], disconnecting from relevant node: Connection reset
[2023-10-27T09:06:26,575][WARN ][o.e.c.c.ClusterBootstrapService] [v-elk-em01.my] this node is locked into cluster UUID [ACjIGY_AT-6yr0tCdySHqw] but [cluster.initial_master_nodes] is set to [v-elk-em01.my, v-elk-em02.my, v-elk-em03.my]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/important-settings.html#initial_master_nodes
next step database v-elk-ed01:
[2023-10-27T00:00:09,286][INFO ][o.e.i.m.MapperService ] [v-elk-ed01.my] [.monitoring-logstash-7-2023.10.27] reloading search analyzers
[2023-10-27T12:19:46,304][INFO ][o.e.t.TcpTransport ] [v-elk-ed01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.206:9301, remoteAddress=/10.1.6.13:34456, profile=default}], disconnecting from relevant node: Connection reset
[2023-10-27T12:53:05,156][INFO ][o.e.t.TcpTransport ] [v-elk-ed01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.206:9301, remoteAddress=/10.1.6.13:53984, profile=default}], disconnecting from relevant node: Connection reset
and ed03 logs:
[2023-10-27T00:00:00,723][INFO ][o.e.i.m.MapperService ] [v-elk-ed03.my] [vcs-logs-2023.10.27] reloading search analyzers
[2023-10-27T11:12:23,716][INFO ][o.e.t.TcpTransport ] [v-elk-ed03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.208:9301, remoteAddress=/10.1.6.14:41618, profile=default}], disconnecting from relevant node: Connection reset
although the cluster status is green.
I don’t know what can be done to restore it to restore normal operation.
Anyone have any ideas?
this is not just my opinion