}
Oct 27 07:01:17 v-elk-lst01.my logstash[42023]: [2023-10-27T07:01:17,681][WARN][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://remote_monitoring_user1:xxxxxx@v-elk-eig01.my:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://v-elk-eig01.my:9200/][Manticore::ClientStoppedException] Connection pool shut down"}
```
Then I started checking the logs on the following Ingest server (v-elk-eig01):
```auto
[2023-10-27T12:48:53,281][INFO][o.e.t.OutboundHandler] [v-elk-eig01.my] send message failed [channel: Netty4TcpChannel{localAddress=/v-elk-eig01.my:54042, remoteAddress=v-elk-ed01.my:9301, profile=default}]: Connection timed out
[2023-10-27T12:48:53,281][WARN][o.e.t.OutboundHandler] [v-elk-eig01.my] sending transport message [Request{indices:data/write/bulk[s]}{30098725}{false}{true}{false}] of size [1533] on [Netty4TcpChannel{localAddress=/v-elk-eig01.my:54042, remoteAddress=v-elk-ed01.my:9301, profile=default}] took [731987ms] which is above the warn threshold of [5000ms] with success [false]
```
v-elk-eig02 logs:
```auto
[2023-10-27T12:04:05,041][WARN][o.e.c.c.ClusterFormationFailureHelper] [v-elk-eig02.my] master not discovered yet: have discovered [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]; discovery will continue using [10.1.5.210:9301, 10.1.5.211:9301, 10.1.5.212:9301] from hosts providers and [{v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] from last-known cluster state; node term 45, last-accepted version 449311 in term 45; joining [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] in term [45] has status [waiting for response] after [14.9m/894401ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/discovery-troubleshooting.html
[2023-10-27T12:04:07,346][WARN][r.suppressed] [v-elk-eig02.my] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:174) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:60) ~[?:?]
at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:35) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:455) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1015) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:981) ~[?:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:469) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:185) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:459) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:292) ~[?:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:146) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:261) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:199) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:379) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$prepareRequest$0(RestMonitoringBulkAction.java:101) ~[?:?]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:96) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:87) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:82) ~[?:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:415) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:544) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:317) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:453) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:549) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:426) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120) ~[?:?]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[?:?]
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
[2023-10-27T12:04:08,602][WARN][o.e.x.m.MonitoringService] [v-elk-eig02.my] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:110) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$1(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:171) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.retry(TransportBulkAction.java:657) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:644) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:513) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
... 33 more
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:640) ~[elasticsearch-8.10.2.jar:?]
... 10 more
[2023-10-27T12:04:09,245][WARN][r.suppressed] [v-elk-eig02.my] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:640) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:513) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
[2023-10-27T12:04:15,041][WARN][o.e.c.c.ClusterFormationFailureHelper] [v-elk-eig02.my] master not discovered yet: have discovered [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]; discovery will continue using [10.1.5.210:9301, 10.1.5.211:9301, 10.1.5.212:9301] from hosts providers and [{v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] from last-known cluster state; node term 45, last-accepted version 449311 in term 45; joining [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] in term [45] has status [waiting for response] after [15m/904405ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/discovery-troubleshooting.html
[2023-10-27T12:04:16,763][WARN][r.suppressed] [v-elk-eig02.my] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:640) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:513) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
[2023-10-27T12:04:17,531][INFO][o.e.t.TcpTransport] [v-elk-eig02.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.6.14:49642, remoteAddress=10.1.5.212/10.1.5.212:9301, profile=default}], disconnecting from relevant node: Connection timed out
[2023-10-27T12:04:17,531][INFO][o.e.t.ClusterConnectionManager] [v-elk-eig02.my] transport connection to [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] closed by remote
[2023-10-27T12:04:17,531][INFO][o.e.c.c.JoinHelper] [v-elk-eig02.my] failed to join {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0} with JoinRequest{sourceNode={v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, transportVersion=8500061, minimumTerm=45, optionalJoin=Optional[Join{term=45, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, targetNode={v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}}]}
org.elasticsearch.transport.NodeDisconnectedException: [v-elk-em03.my][10.1.5.212:9301][internal:cluster/coordination/join] disconnected
[2023-10-27T12:04:18,269][INFO][o.e.c.s.ClusterApplierService] [v-elk-eig02.my] master node changed {previous [], current [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]}, term: 45, version: 449312, reason: ApplyCommitRequest{term=45, version=449312, sourceNode={v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}}
```
next step Master logs v-elk-em03:
```auto
[2023-10-27T10:00:16,238][INFO][o.e.c.m.MetadataMappingService] [v-elk-em03.my] [my-ngfw-logs-2023.10.27/0En_oXU7Rrij4nY_REyT0g] update_mapping [_doc]
[2023-10-27T10:23:20,459][INFO][o.e.c.c.NodeJoinExecutor] [v-elk-em03.my] node-join: [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}] with reason [rejoining]
[2023-10-27T10:28:24,766][INFO][o.e.t.TcpTransport] [v-elk-em03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.212:9301, remoteAddress=/10.1.6.14:54406, profile=default}], disconnecting from relevant node: Connection reset
[2023-10-27T10:42:00,611][INFO][o.e.c.m.MetadataMappingService] [v-elk-em03.my] [my-ngfw-logs-2023.10.27/0En_oXU7Rrij4nY_REyT0g] update_mapping [_doc]
[2023-10-27T11:36:07,447][INFO][o.e.c.m.MetadataMappingService] [v-elk-em03.my] [winlogbeat-srv-sdc-2023.10.27/p1Hi2Sx6SamnWsJrFCBM4A] update_mapping [_doc]
[2023-10-27T11:36:54,642][INFO][o.e.c.c.NodeJoinExecutor] [v-elk-em03.my] node-join: [{v-elk-eig01.my}{pZK4-zUxTzGbfKIcLFVkwg}{BOjq36m9Tsm98rZo4Huxpw}{v-elk-eig01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.10.2}{7000099-8100299}] with reason [rejoining]
[2023-10-27T11:41:52,062][INFO][o.e.t.TcpTransport] [v-elk-em03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.212:9301, remoteAddress=/10.1.6.13:35226, profile=default}], disconnecting from relevant node: Connection reset
2023-10-27T12:04:18,283][INFO][o.e.c.c.NodeJoinExecutor] [v-elk-em03.my] node-join: [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}] with reason [rejoining]
[2023-10-27T12:09:26,847][INFO][o.e.t.TcpTransport] [v-elk-em03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.212:9301, remoteAddress=/10.1.6.14:49642, profile=default}], disconnecting from relevant node: Connection reset
```
next step v-elk-em01:
```auto
[2023-10-27T02:20:31,153][INFO][o.e.t.TcpTransport] [v-elk-em01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.210:9301, remoteAddress=/10.1.6.13:49494, profile=default}], disconnecting from relevant node: Connection reset
[2023-10-27T09:06:26,575][WARN][o.e.c.c.ClusterBootstrapService] [v-elk-em01.my] this node is locked into cluster UUID [ACjIGY_AT-6yr0tCdySHqw] but [cluster.initial_master_nodes] is set to [v-elk-em01.my, v-elk-em02.my, v-elk-em03.my]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/important-settings.html#initial_master_nodes
```
next step database v-elk-ed01:
```auto
[2023-10-27T00:00:09,286][INFO][o.e.i.m.MapperService] [v-elk-ed01.my] [.monitoring-logstash-7-2023.10.27] reloading search analyzers
[2023-10-27T12:19:46,304][INFO][o.e.t.TcpTransport] [v-elk-ed01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.206:9301, remoteAddress=/10.1.6.13:34456, profile=default}], disconnecting from relevant node: Connection reset
[2023-10-27T12:53:05,156][INFO][o.e.t.TcpTransport] [v-elk-ed01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.206:9301, remoteAddress=/10.1.6.13:53984, profile=default}], disconnecting from relevant node: Connection reset
```
and ed03 logs:
```auto
[2023-10-27T00:00:00,723][INFO][o.e.i.m.MapperService] [v-elk-ed03.my] [vcs-logs-2023.10.27] reloading search analyzers
[2023-10-27T11:12:23,716][INFO][o.e.t.TcpTransport] [v-elk-ed03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.208:9301, remoteAddress=/10.1.6.14:41618, profile=default}], disconnecting from relevant node: Connection reset
```
although the cluster status is green.
I don’t know what can be done to restore it to restore normal operation.
Anyone have any ideas?
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [November 1, 2023, 9:03am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/2 "2023-11-01T09:03:00Z")
Does anyone have any ideas? 2 months ago everything worked well, but then at some point errors started. 😭 😭 😭
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 1, 2023, 10:26am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/3 "2023-11-01T10:26:51Z")
What is the output of:
```auto
GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v
```
If some outputs are too big, please share them on [gist.github.com](http://gist.github.com) and link them here.
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [November 7, 2023, 9:36am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/4 "2023-11-07T09:36:20Z")
Hi.
this is what I have:
GET /
```auto
{
"name": "v-elk-ekb02.my",
"cluster_name": "my-elk-prod",
"cluster_uuid": "ACjIGY_AT-6yr0tCdySHqw",
"version": {
"number": "8.10.2",
"build_flavor": "default",
"build_type": "deb",
"build_hash": "6d20dd8ce62365be9b1aca96427de4622e970e9e",
"build_date": "2023-09-19T08:16:24.564900370Z",
"build_snapshot": false,
"lucene_version": "9.7.0",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
},
"tagline": "You Know, for Search"
}
```
GET /\_cat/nodes?v
```auto
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.1.6.75 32 97 1 0.00 0.02 0.00 - - v-elk-ekb02.my
10.1.6.14 31 84 3 0.05 0.09 0.09 i - v-elk-eig02.my
10.1.5.208 39 99 11 0.96 0.97 1.11 d - v-elk-ed03.my
10.1.5.212 23 97 5 0.00 0.00 0.00 m * v-elk-em03.my
10.1.5.211 16 97 0 0.02 0.03 0.00 m - v-elk-em02.my
10.1.6.13 50 84 3 0.06 0.11 0.10 it - v-elk-eig01.my
10.1.5.209 50 97 20 2.17 1.54 1.43 d - v-elk-ed04.my
10.1.5.206 71 98 3 0.84 1.16 1.30 d - v-elk-ed01.my
10.1.5.207 46 99 7 0.99 1.05 1.08 d - v-elk-ed02.my
10.1.5.210 53 97 0 0.00 0.00 0.00 m - v-elk-em01.my
```
GET /\_cat/health?v
```auto
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1699349558 09:32:38 my-elk-prod green 10 4 1981 1252 0 0 0 0 - 100.0%
```
GET /\_cat/indices?v
```auto
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open employeeremotework-2023.01.20 Vw0XJQjOQQqZMsw6X2298Q 2 1 3736 0 1.4mb 755kb
green open aws-s3-ec2-2023.11.04 K7_peZjkTK2TMvtBIVLENQ 1 1 165109 0 187mb 91.9mb
green open aws-s3-ec2-2023.11.05 0WKMjSFtQ1S29DrNxl9jtA 1 1 155546 0 175mb 89.2mb
green open aws-s3-ec2-2023.11.06 xedslMMwRJ68IKTTVz-g4A 1 1 149405 0 170.3mb 85.6mb
green open employeeremotework-2023.01.28 FTy2DgaJSESxV_phsgJahA 2 1 3751 0 1.4mb 731.8kb
green open aws-s3-ec2-2023.11.07 jYRxZ_BQSO2hzZOh8v8dnQ 1 1 59051 0 68.8mb 34.4mb
green open filebeat-8.3.3-cisco-dc-2023.10 p6b09szNQbmOyf1PMbf07Q 2 0 853309 0 328.4mb 328.4mb
green open filebeat-8.3.3-cisco-dc-2023.11 O2RuDqlyQ_SuQjkbuLTpDg 2 0 114719 0 88.3mb 88.3mb
green open fs-winlog-2023.09.01 ulCCjMkZTn2D2T08lI9RBw 2 0 2465577 0 1.6gb 1.6gb
green open .fleet-filedelivery-meta-endpoint-000001 W4hERU9CSGeVjtEhVwnMCw 1 1 0 0 496b 248b
green open fs-winlog-2023.09.02 eKH-bHEuQruAA8TYv4gbgQ 2 0 777236 0 430.8mb 430.8mb
green open fs-winlog-2023.09.04 hngZ5rZBQfOCXRZJ92gKSg 2 0 2847479 0 1.9gb 1.9gb
green open fs-winlog-2023.09.03 8erg6yeuQtuTgO0NNruEIQ 2 0 608512 0 396.2mb 396.2mb
green open fs-winlog-2023.09.06 K5IHj1r1ROucrRPSK19nKg 2 0 2807023 0 1.9gb 1.9gb
green open fs-winlog-2023.09.05 8i5pbyO1QeixEtjLPUyhjw 2 0 13418729 0 6.9gb 6.9gb
green open fs-winlog-2023.09.08 olPM3XYpS5SpK6-sbS4L8g 2 0 4593119 0 2.5gb 2.5gb
green open fs-winlog-2023.09.09 piVNw5GVT8q5V2wrqSiPTw 2 0 16252857 0 7gb 7gb
green open employeeremotework-2022.12.11 L9wSsKRxQdmaB0pGV9xZTg 2 1 30 0 197.1kb 98.5kb
green open fs-winlog-2023.09.07 j-SFB73vT4WOg0Jxo7RXjg 2 0 2837009 0 1.9gb 1.9gb
green open vcs-logs-2023.11.01 b9JRRhG4R2WU-DyPqxn1MA 2 1 229652 0 160.5mb 78.5mb
green open vcs-logs-2023.11.02 ddOUQXPJTmi-bCd5i3oMvw 2 1 281724 0 196.9mb 99.3mb
green open vcs-logs-2023.11.03 PbPdbB0aRVmx5wsbY8vUNA 2 1 315770 0 226.8mb 112mb
green open vcs-logs-2023.11.04 o8f97nyNTbe9Js_i4dlG4Q 2 1 232902 0 167.1mb 82.3mb
green open vcs-logs-2023.11.05 xC63nHajTPOAoolmDXLnwA 2 1 213381 0 151.2mb 75.7mb
green open employeeremotework-2022.12.18 KHMpnYmgSpWHcbu_m27_VA 2 1 60 0 237.3kb 122.7kb
green open vcs-logs-2023.11.07 IGXgFxoiTqe4rEmZ0v_65g 2 1 105400 0 76.2mb 38mb
green open vcs-logs-2023.11.06 Lv_sJJ8qQIGgI4-CHADWUg 2 1 368720 0 260.7mb 130.4mb
green open winlogbeat-srv-login-2023.03 FDjH9LZZQmiqoYEE3k3FtA 2 1 35435071 2223822 28.1gb 14gb
green open winlogbeat-srv-login-2023.04 vhhgP7s2SASNrhB7BUoIkw 2 1 36333207 3049028 29.4gb 14.7gb
green open winlogbeat-srv-login-2023.05 gL_WQH4PQcK-aqjJTzB9tQ 2 1 43066247 4080331 34.5gb 17.3gb
green open winlogbeat-srv-login-2023.06 _om2wKvQSP6mY-i5IpSRHQ 2 1 43584339 2809753 34.6gb 17.3gb
green open winlogbeat-srv-login-2023.07 Wv73qm8kRBSisad1QGvPLQ 2 0 42393665 2818402 16.4gb 16.4gb
green open winlogbeat-srv-login-2023.08 MDUynceESgurWcd2n7urSw 2 0 43855345 398098 16.8gb 16.8gb
green open winlogbeat-srv-login-2023.09 sHP089WUReSflmN4ofP_hw 2 0 43242742 375455 17.1gb 17.1gb
green open winlogbeat-srv-event_code-2023.03-new 0ADF0YsYShGGgndsp7WykQ 2 1 225614 0 400mb 200mb
green open fs-winlog-2023.09.10 JfkvH6U1TTyWMcAoaGjvGg 2 0 774932 0 465.6mb 465.6mb
green open itsm-wan-areas 6bKh-k20Ts-RCT1k9668tA 2 1 847 12139 19.8mb 10.4mb
green open fs-winlog-2023.09.12 CV__OpmKQ3akODid_a6FRw 2 0 8618622 0 3.9gb 3.9gb
green open fs-winlog-2023.09.13 yea4Lc3cSkmuApjSVjvCOQ 2 0 2408777 0 1.5gb 1.5gb
green open fs-winlog-2023.09.14 p7lfMhGLSqig7Q5cTanBpg 2 0 7061848 0 3.8gb 3.8gb
green open fs-winlog-2023.09.15 NZ7WDdCOTnmSPyMaEpbEBw 2 0 2559708 0 1.7gb 1.7gb
green open fs-winlog-2023.09.11 MKZm_mroQwy2PrXuNZyHQg 2 0 13626748 0 5.6gb 5.6gb
green open fs-winlog-2023.09.17 kB88S-wrSA-bvlXm6QZWOA 2 0 591955 0 367.3mb 367.3mb
green open fs-winlog-2023.09.16 S-MzepW_Q2e0GbUXi7j7uw 2 0 669467 0 365.2mb 365.2mb
green open fs-winlog-2023.09.19 HpePK8rpQS2JWKCh-5sMLQ 2 0 16270103 0 8gb 8gb
green open fs-winlog-2023.09.18 8yKLD_9GQaytR5QKE6Qk0Q 2 0 8172718 0 4.8gb 4.8gb
green open winlogbeat-srv-login-2023.10 UuL7ARx-TGChPflglRXk-w 2 0 24653041 135477 9.8gb 9.8gb
green open oseti-channel-mon-oid-2023.11 Lkf1n3HATUOxb2IBMbw9fg 4 1 224 0 655.5kb 253.4kb
green open .fleet-files-agent-000001 kUhiWz30TRGnmWM1iiGgSw 1 1 0 0 496b 248b
green open winlogbeat-srv-sdc-01 yZ7dRefNTPKTzO3BLJcGPA 2 1 0 0 992b 496b
green open fs-winlog-2023.09.20 NGqcDGqyQcO6raxNvl3byw 2 0 2406501 0 1.6gb 1.6gb
green open fs-winlog-2023.09.21 O6xHtsK-QFSOkGQhxJ-vsw 2 0 2778651 0 1.9gb 1.9gb
```
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [November 7, 2023, 9:37am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/5 "2023-11-07T09:37:04Z")
GET /\_cat/indices?v
```auto
green open fs-winlog-2023.09.22 ONaca76hTzKhGfpJG1kwoQ 2 0 2641637 0 1.8gb 1.8gb
green open oseti-sd-wan-2023.10 EXRnu7wMSfO-xUviTFqZbQ 2 1 8473 0 7.1mb 3.8mb
green open oseti-sd-wan-2023.11 TUuZ15GkTIasJZdO7kWyxQ 2 1 4600 0 4.8mb 2.4mb
green open yasno-fin-2023 CSyX_KepScCTomkp8SytGg 2 1 156 0 1mb 530.5kb
green open fs-winlog-2023.09.24 vjNKxklYQIKptuSI5fMAxw 2 0 638278 0 414.1mb 414.1mb
green open fs-winlog-2023.09.23 s-AOFbBfRBCexP3ZXRapGQ 2 0 775514 0 453.9mb 453.9mb
green open fs-winlog-2023.09.26 VQZAfFCxRVe1jEBrab1bKw 2 0 11427168 0 5.3gb 5.3gb
green open fs-winlog-2023.09.25 c0Qvb3OcQPmZ272heUEO-w 2 0 6348160 0 3.8gb 3.8gb
green open fs-winlog-2023.09.28 Ol_7F7ioSQyttLyu7n-l0Q 2 0 3237014 0 2.2gb 2.2gb
green open fs-winlog-2023.09.27 xaaWJhpWTfKUkyGrzkoxNQ 2 0 3164934 0 2.1gb 2.1gb
green open fs-winlog-2023.09.29 qNak5aOOQe6375lUiPuYnA 2 0 5933384 0 3.1gb 3.1gb
green open ngfw-block-2023.04 K9KHdt3hT7CCKcoAy_TWTQ 2 1 1754746 0 1.5gb 782.8mb
green open ngfw-block-2023.05 KI0Z0h5EQxGfMu8RoOMOcA 2 1 2742038 0 2.1gb 1gb
green open ngfw-block-2023.06 Pp7MV5tzTrKdw_JP_wW9Kg 2 1 1745887 0 1.7gb 882.6mb
green open ngfw-block-2023.07 fE1Q1vdqTjqJt396Qlb-Fw 2 1 1045572 0 1006mb 503.5mb
green open ngfw-block-2023.08 nHRUtQqTQg-_UQBuRD5fNQ 2 1 1072145 0 1.1gb 611.4mb
green open ngfw-block-2023.09 rV-3IWkYS3SX8KwF_AZp3w 2 1 1222900 0 1.5gb 755.6mb
green open metricbeat-spdev-2023.11.04 Oo97j9rCRHmSgIZSeuDsDg 2 0 294872 0 124mb 124mb
green open metricbeat-spdev-2023.11.05 bRZJjHwFQOemB86d4j2zoA 2 0 299448 0 129.4mb 129.4mb
green open metricbeat-spdev-2023.11.06 cOamcYtzRT-1mHtfY5hOHw 2 0 299802 0 130.6mb 130.6mb
green open metricbeat-spdev-2023.11.07 aFO2i34QTtKm7TlItRLnIA 2 0 118852 0 52.9mb 52.9mb
green open fs-winlog-2023.09.30 pTHO--2IRTaEYLDCzInC3w 2 0 874508 0 493.7mb 493.7mb
green open employeeremotework-2023.10.06 UdO3uCVBSam0AggKHwWDfw 2 1 2641 0 1.2mb 649.4kb
green open ngfw-block-2023.10 MjUkx_pTQWioAX_KrHHMJA 2 1 5694883 0 4.5gb 2.2gb
green open ngfw-block-2023.11 NJqhMfllSS-eWdd4VI4a5w 2 1 222908 0 257.5mb 130.9mb
green open yasno-rgr-2023.10 ym4VPQNSSneTAdGq92_aNQ 2 1 0 0 1000b 500b
green open yasno-rgr-2023.11 DFhZeQwxSRCf8K5rKGuY3w 2 1 0 0 996b 498b
green open exchange-v-exch-fe05-2023.10 klEZ3dbbRBKYZKwizGyIVQ 2 1 5548441 0 17.5gb 8.7gb
green open exchange-v-exch-fe05-2023.11 KORKTvuOTLe_a_9f-UO9xA 2 1 1061863 0 4.3gb 1.8gb
green open employeeremotework-2023.10.13 NSrwcAGtSNuVKU7cZY9yEw 2 1 2628 0 1.4mb 748.7kb
green open logs-index_pattern_placeholder -y4rpFjoQmCnAwCnXPBJMQ 1 1 0 0 496b 248b
green open winlogbeat-srv-event_code-n-2022.10 hsNMyIqhQb6peGFwcDd4oQ 2 1 172747 0 289.1mb 143.8mb
green open winlogbeat-srv-event_code-n-2022.11 LTMjpOZZTeCI9HVjtfshgQ 2 1 319639 0 534.8mb 267.4mb
green open winlogbeat-srv-event_code-n-2022.12 2RBHKBpbSgK5GJLe6BIKqQ 2 1 182887 0 296.9mb 148.5mb
green open grids-cvp-2023.10 SplfuqEAQF-kzid9LLaOIg 2 1 5573421 0 2.1gb 1gb
green open grids-cvp-2023.11 mnQqznkyRUCyr7b15tX70Q 2 1 2154131 0 920.4mb 463.6mb
green open my-ucsyslog-ucsyslog-gcc-2023.10 O3h2Fqo2QWC9Rb5vgRXKjw 2 1 46660 0 48.2mb 26.9mb
green open my-ucsyslog-ucsyslog-gcc-2023.11 _kW0hZkcSf6GLbiDXcs1qg 2 1 9405 0 10.1mb 5.1mb
green open employeeremotework-2023.10.20 K9_Za5MZRAWPlt9Gee6Naw 2 1 1329 0 1011.5kb 453.5kb
green open employeeremotework-2023.10.27 ovFi3GEIREGmkiQQJktzLQ 2 1 1331 0 834.3kb 417.1kb
green open employeeremotework-2023.06.02 C9kJQMlST5i-s9ThjyPuSQ 2 1 2556 0 1mb 523.1kb
green open aws-s3-signin-2023.11.04 -AHrADZGQ_6WhbJoxtksuQ 1 1 68524 0 67.4mb 33.7mb
green open .internal.alerts-security.alerts-default-000001 H5r8lZonRTi_ceH4IFk_vg 1 1 0 0 496b 248b
green open aws-s3-signin-2023.11.05 FosrdVSaS1igSCLHdMfiMA 1 1 66330 0 65.9mb 32.9mb
green open employeeremotework-2023.06.09 Wi9yMGr9RuayA_4YP6ggHw 2 1 2560 0 1mb 570.4kb
green open aws-s3-signin-2023.11.06 3ZavkuiATkOCqR-bY7nNWg 1 1 64609 0 63.9mb 32.1mb
green open mfa-token q6mHL0EdS0iHGWg5Lnvx4Q 2 1 1 0 45.6kb 22.8kb
green open aws-s3-signin-2023.11.07 mBZ6Gcb0ToCOeZTwsUI9Xw 1 1 9503 0 11.1mb 5.5mb
green open winlogbeat-srv-user-workstation-2022.12 yZej15q2RBqD07HRn6TTMw 2 1 699 242 1.3mb 704.7kb
green open wifi-dc1-2023.10 fLxMeOgHQkiW4mIQ2-X57A 2 0 4072142 0 1.2gb 1.2gb
green open wifi-dc1-2023.11 iHpSCQyqRdOiHuHwKOAXIQ 2 0 867207 0 312.1mb 312.1mb
green open ups-syslog-2023 y4vGyL5UTTG1Stmkl0mn2Q 2 1 1866370 0 685.3mb 344.7mb
green open employeeremotework-2023.06.16 2s5YKo6UTHi4U9At-qDqEQ 2 1 2564 0 1mb 552.4kb
green open ups-data-1999 gfX9DUViTnGpKaoXDQAhoQ 2 1 65 0 305.1kb 145.8kb
green open employeeremotework-2023.06.23 op2eZYENTSSD5LAygaIS4w 2 1 2587 0 1mb 552.5kb
green open employeeremotework-2023.02.04 XYEa0UQ7SlmRVE-AjOD3dQ 2 1 3741 0 1.3mb 734.2kb
green open my-ucsyslog-2023.10 AAvpVTJ8SwqSEBRKwjGZ8w 2 1 54768146 0 20.9gb 10.4gb
green open my-ucsyslog-2023.11 GTQ0J6zlS3-_iFmyPNTVLg 2 1 10988054 0 4.5gb 2.1gb
green open azurevpn-device-2023.04 LTVPdPrrTsq-H-rARXRD0Q 2 1 4 0 45.5kb 22.7kb
green open sftp-logs-2022.12 xMnwKGZtSGKyaC9GgU08tg 2 1 10360959 22 3.5gb 1.7gb
green open azurevpn-device-2023.05 LvYApjPPSgqICPnp3f-Fwg 2 1 4 0 74kb 37kb
green open employeeremotework-2023.06.30 krGsYDvQQCSdNt1ncEIYIA 2 1 2576 0 1mb 525.4kb
green open employeeremotework-2023.02.11 lGdDP6Z_Rl619gCEslw3SA 2 1 3732 0 1.3mb 674.5kb
green open employeeremotework-2023.02.17 WAOXMTcpQmm2ygllB79ijg 2 1 3739 0 1.3mb 672.6kb
green open my-uccxsyslog-2023.10 S6I33IUNT5qUvbcpZ4jtmg 2 1 8926 0 5.5mb 2.6mb
green open my-uccxsyslog-2023.11 3enTDlOUSLaq779dmwqbCA 2 1 2108 0 1.6mb 772kb
green open oid-ups-2023 V_fIs2EqSqq-PxKNhtGlng 2 1 276 508 1mb 561.4kb
green open grids-cups-2023 snLx9lWQTPWo3IPp66TejQ 2 1 335 0 567.2kb 268.4kb
green open winlogbeat-exchange-2023.11.01 UKctnICXTMaf_IyU7ZGihw 2 1 12093 0 31.4mb 10.7mb
green open winlogbeat-exchange-2023.11.03 dMJFnU13RWCE29VB7ac1Uw 2 1 12595 0 35.1mb 12.8mb
green open winlogbeat-exchange-2023.11.04 ZnpVIJ3uQaueiJVHDc7mZQ 2 1 11117 0 29.1mb 10.3mb
green open grids-fin-2023 A1yfoDCJSkivUnLC4dnU9g 2 1 63 0 1.2mb 683.9kb
green open winlogbeat-exchange-2023.11.05 PoXj5bMXTWmot0ASuLuItQ 2 1 10872 0 26.9mb 9.2mb
green open winlogbeat-exchange-2023.11.02 gjCm4de5QhyytmWJGRXTHg 2 1 12186 0 33.6mb 12.5mb
green open winlogbeat-exchange-2023.11.07 kPe8dBsKRLq0bbr0kT3Dvg 2 1 4861 0 13.5mb 5.7mb
green open winlogbeat-exchange-2023.11.06 XpE3tok0Tw2oTSvyo_rISQ 2 1 12212 0 33mb 11.7mb
green open exchange-v-exch-fe06-2023.10 KCFHkcG5S7iJUoufs7uyVA 2 1 5467424 0 17.3gb 8.7gb
green open exchange-v-exch-fe06-2023.11 H0u3Q-4cTXOIE1SkDwls8A 2 1 1081789 0 3.8gb 1.9gb
green open cvp-ycc-activity-2023.11 e0wPiYsOTCy1ORi7hL7zXQ 2 0 24922 0 6.9mb 6.9mb
green open cvp-ycc-activity-2023.10 NTmVQZMRSTq-dvnt4HSXIQ 2 0 43236 0 18.5mb 18.5mb
green open exchange-v-exch-fe02-2023.10 FgNmjczjSVCuiuNGUMP7CQ 2 1 6956295 0 22.2gb 11gb
green open exchange-v-exch-fe02-2023.11 QRIcAknzTcGosA6W0q0yUA 2 1 1643534 0 6.5gb 2.8gb
green open winlogbeat-srv-user-workstation-2023.04-new mB0Lv2-MTKyblnSnC9H31Q 2 1 5243353 0 3.2gb 1.6gb
green open employeeremotework-2023.02.24 kc0J8QNTQwaxGzWHfyHdNg 2 1 2494 0 1mb 534.9kb
green open filebeat-nexus-2023.10 26aOUfS4SdG7ErAcVTT32w 2 0 4434 0 2.5mb 2.5mb
green open filebeat-nexus-2023.11 nV_YDgJuRziudbrYmNUsMg 2 0 762 0 581.5kb 581.5kb
green open .internal.alerts-observability.slo.alerts-default-000001 XcSehwDhTHeNtDofRmwGdg 1 1 0 0 496b 248b
green open fs-winlog-2023.10.01 FwKvWixEQaaF86UDXUoOgw 2 0 869672 0 568.9mb 568.9mb
green open fs-winlog-2023.10.02 n1l9-YlSTNOZp-eShZ8TzA 2 0 9972987 0 5.8gb 5.8gb
green open fs-winlog-2023.10.03 dJtl4ae6S72SuaA3xpBIGg 2 0 9802099 0 4.3gb 4.3gb
green open fs-winlog-2023.10.04 32DqDN67QHyea8mACgxQ4Q 2 0 2512000 0 1.6gb 1.6gb
green open fs-winlog-2023.10.05 BsOYUOf0QiKF80Z9HpW1eg 2 0 3597750 0 2.2gb 2.2gb
green open fs-winlog-2023.10.06 gNZSIoDERTWFGHzurJnjFg 2 0 2883957 0 1.8gb 1.8gb
green open fs-winlog-2023.10.07 FzSoGhYmQraCumOHoWTJUw 2 0 1110094 0 597.4mb 597.4mb
green open fs-winlog-2023.10.08 FbA98MqYRzqe0zYzy5EyVw 2 0 796803 0 487.5mb 487.5mb
green open fs-winlog-2023.10.09 nLjT6uY2QIyzUEiSASik7Q 2 0 2856350 0 1.9gb 1.9gb
green open expw-syslog-2023.11.01 2dnSaXXZQgmMgHE3FL8nug 2 1 1338998 0 1.1gb 592.2mb
green open expw-syslog-2023.11.02 Rkd7JyzGRfea8H9cUutlCg 2 1 1326857 0 1.1gb 584.6mb
green open expw-syslog-2023.11.03 NgTLyCYSRIeC292deUcFsg 2 1 1343268 0 1.1gb 597.7mb
green open expw-syslog-2023.11.04 KfaC1dVWRh-8SVh3tCUxbA 2 1 1287224 0 1.1gb 569.3mb
green open expw-syslog-2023.11.05 GTT1uM0CSKeELrt0RPAPcg 2 1 1290246 0 1.1gb 571.2mb
green open expw-syslog-2023.11.06 a60HtyCiQdWThBI5ngkmCg 2 1 1329722 0 1.1gb 587.7mb
green open expw-syslog-2023.11.07 DMB4pW4mS8W_nEXkU9LkyA 2 1 528193 0 479.3mb 239.7mb
green open winlogbeat-srv-user-workstation-geo-2023.04-new JVh0rCf4S-uFofko1lxcVQ 2 1 5243445 0 3.4gb 1.7gb
green open index OSJN1F-2TIaoSAh0pashSA 1 1 1 0 6.1kb 3kb
green open systemd-uc-2023 9QMKGycfSEatGiSVgmzaKw 2 1 706 0 1mb 532.7kb
green open fs-winlog-2023.10.10 S6ElM1xGSK6qWIaTqqrmxQ 2 0 9577450 0 4.1gb 4.1gb
green open fs-winlog-2023.10.11 9ZmLS5nwQJmARG3fxCPA5A 2 0 2938859 0 1.9gb 1.9gb
green open fs-winlog-2023.10.12 F0HeLtx_SPm5yzb5AEWc0g 2 0 2744834 0 1.8gb 1.8gb
green open fs-winlog-2023.10.13 tatnVJs5R4yDRz6q7Xap8g 2 0 4475691 0 2.6gb 2.6gb
green open fs-winlog-2023.10.14 fbL_ybfvSQmJf0En7ULSTA 2 0 9499474 0 4.1gb 4.1gb
green open fs-winlog-2023.10.15 seM0wAb3SNWNweISMpo8VA 2 0 679973 0 422.6mb 422.6mb
green open fs-winlog-2023.10.16 yVC1ZXcBSTKHOwmuhCvDTA 2 0 2375819 0 1.6gb 1.6gb
green open my-ngfw-logs-2023.11.07 6pvJy6tQR7C0HccEK4AlDg 2 0 14511603 0 9.8gb 9.8gb
green open cc-winlogbeat-2023.11.01 Be_AkdwwSVeOvnLtjWq0bg 2 1 2771 0 3.9mb 2.4mb
green open cc-winlogbeat-2023.11.02 eWqcSLklTxi0eozRphkHTA 2 1 1658 0 4.1mb 1.9mb
green open cc-winlogbeat-2023.11.03 SCfXa76MRKig0f2cKqX1IA 2 1 742 0 2.1mb 1.2mb
green open aws-s3-s3-2023.11.04 z_vLhFLcRB2uZCCCWnmv2g 1 1 218334 0 350.8mb 173.1mb
green open aws-s3-s3-2023.11.05 odnixw_zSr67fZljl3rYyw 1 1 210217 0 333.6mb 167.1mb
green open aws-s3-s3-2023.11.06 onLtJIuxTN-N7sA8RQTICA 1 1 180261 0 296.7mb 147mb
green open aws-s3-s3-2023.11.07 5Xs6ug2zQgqtDAIRJg_DUg 1 1 81143 0 131.5mb 64.5mb
green open cc-winlogbeat-2023.11.06 jXTCc1OzRmGtlwRvfKw6iA 2 1 334 0 1.5mb 565.8kb
green open cc-winlogbeat-2023.11.05 8zIKx-VrSFKsoSM24LB4QQ 2 1 141 0 1.1mb 452.4kb
green open cc-winlogbeat-2023.11.04 r5FB_-6PQoGccDogV-63pA 2 1 143 0 1mb 398.8kb
green open cc-winlogbeat-2023.11.07 AbQ5p9NBSj2jRLVm_pnLrw 2 1 45 0 1011.8kb 322.4kb
green open fs-winlog-2023.06.07 m9sUPawrSVSOFnSqPZvEAg 2 0 748589 0 530.7mb 530.7mb
green open fs-winlog-2023.06.08 P9iRMW2LT6CVgqMMBaaGEw 2 0 2235461 0 1.5gb 1.5gb
green open fs-winlog-2023.06.09 G0vaent4Rx61OwvfN-SIeg 2 0 1949826 0 1.2gb 1.2gb
green open winlogbeat-srv-device-w-2023 Ub-zsd3LQCCuCqPyD4vfgw 2 0 13795 57807 56.7mb 56.7mb
green open aws-s3-all-2023.11.04 aEXubqCJSUWxBBzuFcPsbw 1 1 698855 0 899.2mb 445.5mb
green open aws-s3-all-2023.11.05 ew0RM-PWQmi1Z9cOVJqq6w 1 1 623294 0 817.2mb 406.6mb
green open aws-s3-all-2023.11.06 tikyqJ3WTk-wKd4VY5z8mg 1 1 625314 0 811.2mb 415.9mb
green open aws-s3-all-2023.11.07 om7byGtQQcepZMRE7xe4VA 1 1 240788 0 311mb 157.7mb
green open .internal.alerts-observability.logs.alerts-default-000001 IL5tlMTKSYeSYM2wHBC5IA 1 1 0 0 496b 248b
green open .fleet-files-endpoint-000001 H9CzhvqPR5qOxQnX2Uf8RA 1 1 0 0 496b 248b
green open employeeremotework-2023.11.03 1S5749JsTBO6s3xxnAMBeQ 2 1 1327 0 856.6kb 326.6kb
green open fs-winlog-2023.06.10 8cFrNGIISlaBMTk1IWeQKg 2 0 897672 0 487.4mb 487.4mb
green open fs-winlog-2023.06.11 u9u7dWuyQJeDQYv0oIMXRg 2 0 499880 0 320.2mb 320.2mb
green open fs-winlog-2023.06.12 SiRCsurJT32-nojbyJ2Hjg 2 0 2138835 0 1.5gb 1.5gb
green open fs-winlog-2023.06.13 8zz3PW__Tea39PJUburOGg 2 0 9925070 0 4.3gb 4.3gb
green open fs-winlog-2023.06.14 R6t2YtACT-qmzPZRVIuD_Q 2 0 2224637 0 1.4gb 1.4gb
green open fs-winlog-2023.06.15 BSJW-D_6SUSugflkNF9LBQ 2 0 1856479 0 1.2gb 1.2gb
green open fs-winlog-2023.06.16 JkOyKJ1cT6akOnOaNvNpSA 2 0 4051908 0 2.3gb 2.3gb
green open fs-winlog-2023.06.17 hli0-BnkSQOj01Nu18M15Q 2 0 849281 0 426.2mb 426.2mb
green open .internal.alerts-observability.apm.alerts-default-000001 ZUcQNIm5Ta2DuC4egsKeTg 1 1 0 0 496b 248b
green open fs-winlog-2023.06.19 iEf6BAd-QtWyAPwCyxezqA 2 0 2023001 0 1.3gb 1.3gb
green open fs-winlog-2023.06.18 r9pHfj9XQiywSvXQQjYDmg 2 0 546515 0 336.5mb 336.5mb
green open winlogbeat-srv-event_code-n-2023.02 lv7MTpSTReKAdwjs54DAKg 2 1 47953 0 86.2mb 43.1mb
green open winlogbeat-srv-event_code-n-2023.01 TtNc_NFBTyCTXFsLMAyzQA 2 1 106191 0 186.3mb 93.2mb
green open fs-winlog-2023.11 5_TLDW0gSZqB6GdnNUqWhw 2 0 113389583 0 54.4gb 54.4gb
green open fs-winlog-2023.10 W4jzY_cUTn2gjyXTt71JtQ 2 0 37280471 0 21.8gb 21.8gb
green open scom-srv-performance-2023.09 NvduYfRsSJ-GZfK6UGNoAg 2 0 32952455 0 6.6gb 6.6gb
green open scom-srv-performance-2023.08 J0yJHpM1TgSKL9TVOymwFg 2 0 37348833 0 7.5gb 7.5gb
green open winlogbeat-srv-login-2023.02_1 PltgV9CBQwmmDCqSCMOnmw 2 0 23366393 0 8.9gb 8.9gb
green open yasno-vvb-2023 dx96HjrLQ2iFFgF-q1wJIw 2 1 401 0 1.8mb 1mb
green open scom-srv-performance-2023.07 Xr3Y-C9DQBOZ6i2TbCpl7g 2 0 39961146 0 7.8gb 7.8gb
green open scom-srv-performance-2023.06 IM9kZ1PsTemsC5p47Th8JQ 2 0 12547091 0 2.2gb 2.2gb
green open fs-winlog-2023.06.20 W9W2BzclTRCyzzqBHXmUPA 2 0 9677423 0 4.5gb 4.5gb
green open fs-winlog-2023.06.21 8CAFxt47QWCR1L25ALNmow 2 0 1951529 0 1.2gb 1.2gb
green open fs-winlog-2023.06.22 2VFZger6TVOsZpSL8GZEcQ 2 0 3896515 0 2.2gb 2.2gb
green open fs-winlog-2023.06.23 Gaxhdxn7TB-3rNHjXhWqJw 2 0 1459892 0 903.3mb 903.3mb
green open fs-winlog-2023.06.24 UUeDB2s_TV-GfL0UrtPJ4g 2 0 624266 0 317.9mb 317.9mb
green open fs-winlog-2023.06.25 qb1oXcVKRM69FMKqe10yFQ 2 0 474090 0 304mb 304mb
green open fs-winlog-2023.06.26 -25yTIUWSju4xLS_kd7A0Q 2 0 2161968 0 1.5gb 1.5gb
green open fs-winlog-2023.06.27 QnHK0ReJQFyWs_RAhB__gQ 2 0 14413452 0 6.7gb 6.7gb
green open fs-winlog-2023.06.28 GAJB63ZzRueCVFm4u6JyDg 2 0 1982918 0 1.2gb 1.2gb
green open winlogbeat-srv-user-duration QCQ_S7A8Q6qnFlGogfwOwg 1 1 3109251 174061 902.4mb 451.2mb
green open fs-winlog-2023.06.29 O-yCIg40QS-TMo211HzHkQ 2 0 2170710 0 1.4gb 1.4gb
green open exchange-v-exch-fe03-2023.10 zgF-SMK_RBi04msQA8jy0w 2 1 6450395 0 20.1gb 10gb
green open exchange-v-exch-fe03-2023.11 IQEHCEBfScGMoHHw5jDOhw 2 1 1517011 0 6.8gb 2.7gb
green open winlogbeat-srv-user-workstation-2023.01 4Zmj1L7NT4KGWRyLYI8uqA 2 1 4554020 896531 3.1gb 1.6gb
green open winlogbeat-srv-user-workstation-2023.02 y7srtbWdQ4eWvT1pEQbAwg 2 1 11026928 2566625 8.1gb 3.9gb
green open scom-srv-performance-2023.11 JsuLPyXmRBCsiZKWgIfnFg 2 0 7321012 0 1.4gb 1.4gb
green open winlogbeat-srv-user-workstation-2023.04 sYawpa6-TYO5uPLyF6Ufxg 2 1 7234648 1912651 5.4gb 2.7gb
green open winlogbeat-srv-user-workstation-2023.05 -6j7VQj-RheGMvR832Z8Wg 2 1 15424958 3107644 11gb 5.4gb
green open winlogbeat-srv-user-workstation-2023.06 DHzbJ8T5Si2S1shVqUUWxQ 2 1 15699927 3784506 11.4gb 5.6gb
```
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [November 7, 2023, 9:38am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/6 "2023-11-07T09:38:33Z")
...
```auto
green open winlogbeat-srv-user-workstation-2023.07 PKovj8v0S86AHDWsKZMTTw 2 1 15539620 3539669 10.9gb 5.4gb
green open winlogbeat-srv-user-workstation-2023.08 egIZ17DsQo6B-h0zhRWVbw 2 1 11128039 1541626 7.6gb 3.7gb
green open winlogbeat-srv-user-workstation-2023.09 HGzurQbJQJ6Wn4oAW9JcbQ 2 1 4189021 43275 2.9gb 1.4gb
green open scom-srv-performance-2023.10 MW6-juAoRQWJI_jkdzHmPA 2 0 25117579 0 5gb 5gb
green open fs-winlog-2023.06.30 3UsVhRBTRnq0khHSbMKsJQ 2 0 2272394 0 1.2gb 1.2gb
green open itsm-srv-service ay64J57nTdmN9mn_GwM43g 2 1 1783 5669 5.5mb 2.8mb
green open employeeremotework-2023.07.07 aTMxpNsNT8anNPc9mA1T2A 2 1 2535 0 1mb 518.1kb
green open winlogbeat-srv-user-workstation-2023.10 YWufxikdQ0O07YPJJs3jfA 2 1 2361719 43185 1.6gb 863.5mb
green open employeeremotework-2023.07.14 kNvuL_4ITqW7oTfsgBSTFQ 2 1 1373 0 696.3kb 339.1kb
green open sftp-logs-2023.01 PwRUKZr0RMWmtii4LroGFw 2 1 10064777 26 3.3gb 1.7gb
green open sftp-logs-2023.02 Y4Ny_qqHQ5istriQISTfdw 2 1 9855123 0 3.3gb 1.6gb
green open sftp-logs-2023.03 t1rSbczXT0W7Ps3Mlapc5Q 2 1 11762030 0 3.8gb 1.9gb
green open sftp-logs-2023.04 t3gM06rlRJm9W1LxvGEwyg 2 1 11231378 0 3.3gb 1.6gb
green open sftp-logs-2023.05 uOAHAZWQTh6K-lTbTUfmdQ 2 1 6301206 0 1.9gb 1009.3mb
green open sftp-logs-2023.06 z4V83T-CTaiGm7J8c19OXA 2 0 6089367 0 993.2mb 993.2mb
green open sftp-logs-2023.07 3Gr7Rd1GQe277v5hiq8nyQ 2 0 6613160 0 1gb 1gb
green open sftp-logs-2023.08 ToN2pYcbS4icP7Pjlgw4eA 2 0 6107103 0 996.6mb 996.6mb
green open sftp-logs-2023.09 22s-aNhAQpmBMnB2R32kgw 2 0 5659678 0 908.8mb 908.8mb
green open cvp-ycc-app-2023.09 p9mprj43RkG7PGMxQG75jA 2 0 155 0 244.4kb 244.4kb
green open rdweb-info-2023 GnKGFSWBSDGnZpxABZEJ4Q 2 1 20693842 0 11gb 5.5gb
green open rdweb-info-2022 uNnx60CCS3moypM8TLj_UA 2 1 3525513 0 1.7gb 913.2mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.06 mWTLez5sTZiTENOHYwhumA 2 1 5073435 0 1.8gb 960.7mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.07 mv7XMqM8TLmgWAUZXU_MaQ 2 1 7522640 0 3gb 1.5gb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.08 HCze42OXQ0iMdsJV1JVvfQ 2 1 7026437 0 2.6gb 1.3gb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.09 1iS9TpLtSE2IYhQth3grSA 2 1 2884175 0 1gb 525.6mb
green open tmp-cisco-ios-1-2023.05 c_-CKn2kR1GB1g68QVtIeA 2 1 8899 0 4.6mb 2.3mb
green open tmp-cisco-ios-1-2023.06 6zI-eU8aSGCPSPHx3hypdg 2 1 131212 0 46.4mb 23.2mb
green open tmp-cisco-ios-1-2023.07 xF8WxT-xSrSjxE8PcNwaJA 2 1 17487 0 7.7mb 3.9mb
green open employeeremotework-2023.03.03 k7XUgfvyS72OsfTXK1WCug 2 1 2492 0 1mb 516.8kb
green open tmp-cisco-ios-1-2023.08 tl69kfD_Tlu-BMUCXJYe3w 2 1 14162 0 7mb 3.5mb
green open tmp-cisco-ios-1-2023.09 -up89wbrS02rH92cEctnLQ 2 1 10650 0 4.7mb 2.3mb
green open employeeremotework-2023.07.21 2YMVdmy9TJyjZNEgmUkTNg 2 1 1384 0 703.4kb 351.7kb
green open cvp-ycc-app-2023.10 fQCgjXudT8-9TJNz6NisCQ 2 0 320 0 326.5kb 326.5kb
green open employeeremotework-2023.07.28 hWksRHrEQRue918_t6M41w 2 1 2606 0 1.1mb 573kb
green open sftp-logs-2023.10 bVOFrV1gT_C76UEDRGc9Ww 2 0 5705255 0 918mb 918mb
green open sftp-logs-2023.11 uHLvFVTsRduCHIXpqbyBrQ 2 0 1228888 0 211.1mb 211.1mb
green open azurevpn-olwayson-2022.07 Or8uU_xLTbieftwReFCg6Q 1 1 1258317 0 938.5mb 469.2mb
green open .monitoring-logstash-7-2023.11.01 UaZeoHk6TkS8rJq7vx8Ozg 1 1 8891062 0 2.5gb 1.2gb
green open .monitoring-logstash-7-2023.11.02 LwPrVvhBTrO_un2vEBrcPQ 1 1 8395888 0 2.4gb 1.2gb
green open .monitoring-logstash-7-2023.11.03 gGpaSc-kR9KdDgOc-PrLrA 1 1 8726519 0 2.5gb 1.2gb
green open .monitoring-logstash-7-2023.11.04 90hPllLdRc69_i2ezasDBQ 1 1 8996919 0 2.6gb 1.3gb
green open .monitoring-logstash-7-2023.11.05 R5z2VaNVRzm2GThIpCfSIw 1 1 8721658 0 2.5gb 1.2gb
green open ups-data-2000 1LAoGPl3TG6U7UIrN2gGHg 2 1 11812 0 6.2mb 2.8mb
green open grids-rgr-2023.11 fYh3dwPoQwG3gTaas7WUug 2 1 1935 0 2.3mb 1.2mb
green open grids-rgr-2023.10 9tJKVYWJR_i_53VaGhy9MA 2 1 7426 0 4.7mb 2.2mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.14 djW-QwpETiqLI37pbE4How 2 1 3732690 0 1.5gb 771mb
green open tmp-cisco-ios-1-2023.10 Ver0L45qQN-kGiG9mP6ohg 2 1 42285 0 28.9mb 13.9mb
green open tmp-cisco-ios-1-2023.11 O_p1b4r1RZqCqWZ5v5yNuQ 2 1 12505 0 7.4mb 3.2mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.15 mnTNphesRoGoVkqh-hMzqQ 2 1 2566142 0 1.1gb 577.9mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.16 HanpABbtSLmH8OJ3B5dcjQ 2 1 2572999 0 796mb 400.4mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.17 iRarOzhsSjedKBVNk5ENQA 2 1 3291964 0 1.1gb 599mb
green open employeeremotework-2023.03.10 Ftrb8V9TTAum2NsKsXhEMQ 2 1 3280 0 1.2mb 648.1kb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.18 LrDg3VcNRMugV4IBXI8ENQ 2 1 4628283 0 1.8gb 932mb
green open user-location QrG31Ft6SqKl4xC1q99nAw 1 1 0 0 496b 248b
green open filebeat-erp-aud-v-sap-perpn10-2023.09.19 GzV5k7QYSWShQpyDzKeQew 2 1 9152067 0 3.2gb 1.6gb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.10 Y1UW_1JfRXK0VOcrTnRdfA 2 1 972565 0 417mb 208.6mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.11 3c04PTdKSoKZCBgE232QVA 2 1 17133499 0 6.2gb 3.1gb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.12 QeUNveq6RbWWYLa56okCyg 2 1 10229953 0 3.7gb 1.8gb
green open employeeremotework-2023.03.17 xtbe-m7VQJWn6jo0ro_AsA 2 1 2503 0 1mb 527.6kb
green open azurevpn-olwayson-2022.12 bQS88qDTR66PvFMlOFgk0w 2 1 19290621 0 20.7gb 10.3gb
green open cvp-gcc-activity-2023.10 9ZAcK0OqSCSgPQuC2eZssQ 2 0 412288 0 190.2mb 190.2mb
green open cvp-gcc-activity-2023.11 04DcTVn8TVeDWBH2OH4f4Q 2 0 198445 0 80.1mb 80.1mb
green open metricbeat-fs-2023.11.03 XJJJd88rQamT9sBF1w0ojw 2 0 1045246 0 373mb 373mb
green open metricbeat-fs-2023.11.02 rj1E5OTRQ9CSjzMoLLGE7Q 2 0 1053478 0 378.4mb 378.4mb
green open metricbeat-fs-2023.11.01 ZSo3QLnpT1uZeKHNys1snw 2 0 1045322 0 378.4mb 378.4mb
green open metricbeat-fs-2023.11.07 Te1WQBGGQ4yXTLCPRhN2DQ 2 0 419365 0 150.8mb 150.8mb
green open metricbeat-fs-2023.11.06 mum9r1uWSdSYqvh-s-3hVQ 2 0 1038182 0 372.1mb 372.1mb
green open metricbeat-fs-2023.11.05 OubeVGMnRnCtfyx37mPfNQ 2 0 1038037 0 369.9mb 369.9mb
green open metricbeat-fs-2023.11.04 nhA7dSNGS1yqifUMPh63kA 2 0 1038124 0 369.8mb 369.8mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.13 OEtwbMqgSiKET0Hffuwm0A 2 1 3472200 0 1.3gb 671mb
green open .monitoring-logstash-7-2023.11.06 t-b2GFjRRAyRRZTIL3wR_g 1 1 8478104 0 2.4gb 1.2gb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.20 cdUzRmqRToeSEJ8Qz7bPMw 2 1 11276152 0 3.8gb 1.9gb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.21 ueTtQDVkQ5KTJUFM5XVyIw 2 1 13422143 0 4.9gb 2.4gb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.22 gzb2Xq0RQqWV0uTWRYYrqg 2 1 11685141 0 4.3gb 2.1gb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.23 8_Me9ZXKTpylf2ZjdCd_sA 2 1 1867622 0 637mb 318.5mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.24 GWy9ydnqTpqtxHg8d-NBvg 2 1 911568 0 404.3mb 203.1mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.25 _p9vUr5ISyab-dKlxDm0Wg 2 1 5276428 0 2gb 1gb
green open doc-aws-metric-test-2023.10 rc3Z5RTbT4CRt-G3y5K4Aw 2 1 109059 0 105.4mb 49.2mb
green open doc-aws-metric-test-2023.11 2lD-0v0mRQW3HZ1q2BcBVw 2 1 20266 0 28.1mb 12.9mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.26 MnaoZrR5Sk2Xw6wenYujyQ 2 1 4624547 0 1.7gb 919mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.27 LYRg0UIgR0aoxJFzfDdBNQ 2 1 3531605 0 1.4gb 758.8mb
green open filebeat-sap-v-sap-webpen1-2023.10 MzsAVMZ8QW68HWABMrhj-w 2 0 11825514 0 2.7gb 2.7gb
green open filebeat-sap-v-sap-webpen1-2023.11 wtKePkwARwWQtDFfTajk9g 2 0 3697043 0 845.1mb 845.1mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.28 vqD1vlVkRr-8zQurVe0olw 2 1 4281778 0 1.8gb 939.2mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.29 Gffr_01URcKM0H8JHW85Sg 2 1 4638140 0 1.9gb 1017.9mb
green open .monitoring-logstash-7-2023.11.07 Fmd8S277S6CXBibUBePFyA 1 1 3586734 0 1.1gb 588.1mb
green open employeeremotework-2023.03.24 FLx3AeCzRHqZjOJUACA2yw 2 1 2487 0 1mb 537.2kb
green open metricbeat-2022.10.10 GNPGTXLmSWquMiUhgkSILw 1 1 4228042 0 3.4gb 1.7gb
green open ftp-logs-2022.12 R53jyVeFS7aWs04y5egJiA 2 1 72438239 0 24gb 12gb
green open metricbeat-2022.10.11 DsGjzZEpQ-OeS7SIuiPY2g 1 1 2261235 0 1.8gb 953.6mb
green open metricbeat-2022.10.08 5Fg0q_vpT8qlOCIk4ykyhw 1 1 38549 0 37.9mb 18.9mb
green open metricbeat-2022.10.09 83QA0gj9TOmFYIvUcQRl9A 1 1 1034466 0 905.6mb 452.8mb
green open filebeat-erp-aud-v-sap-perpn10-2023.09.30 y5SRUQchQhKVeqKuMq72wg 2 1 4050784 0 1.6gb 831mb
green open oss-sap-ivr_grids-2023.10 _aSqWtRcQdiQ9nShBUwFgg 2 1 1126090 0 651.3mb 325mb
green open oss-sap-ivr_grids-2023.11 WDCbNL_CQYeBX6U4gm_YTw 2 1 650310 0 312.2mb 150.6mb
green open ups-data-2021 mger58zEQKSQ7qSLVxO_MA 2 1 4500 0 4.1mb 2mb
green open ups-data-2022 FRBHwdMlT-C1SNfo8XmdKg 2 1 60917 0 50.5mb 25.4mb
green open ups-data-2023 YjHVBVnBTdCbGObvEA-Kzw 2 1 18383114 0 9.6gb 4.8gb
green open employeeremotework-2023.03.31 P9smQNTOR3ayUlTUR1DojQ 2 1 2495 0 1.1mb 599.5kb
green open grids-vvb-2023 R-IGjm6CRseFWPvsF3Ebxg 2 1 236 0 992.2kb 491.5kb
green open o365-update-health LhGawCjNR5arNEH8DOhmPA 2 1 44 52 240.1kb 120kb
green open cc-sm-2023.10 F0OCa2GyQI6GuliyjkfP_w 4 0 34571 0 23mb 23mb
green open cc-sm-2023.11 lNVtCDeaR3Sk-wPJdhWaEA 4 0 42080 0 17.5mb 17.5mb
green open .internal.alerts-observability.metrics.alerts-default-000001 _evXQzVlS3mcJnRJI_pIMw 1 1 0 0 496b 248b
green open .fleet-filedelivery-data-endpoint-000001 wwfhJMubQOyoQCLMXrePTg 1 1 0 0 496b 248b
green open test-itsm-users rzL7aeXhTdK-q186g4sHyA 1 1 98878 9150 142mb 71mb
green open my-ucsyslog-gcc-2023.10 jZQ3BDf4SE-W6UqsCvrvJA 2 1 1129 0 3.4mb 1.9mb
green open fs-winlog-2023.07.01 KYIV9IK9Roe4YQKwB00x5Q 2 0 1460800 0 695.3mb 695.3mb
green open fs-winlog-2023.07.02 dID1ASvDTGis7DJNB64PcQ 2 0 1389808 0 640.4mb 640.4mb
green open fs-winlog-2023.07.03 qHMS0PrfTM25lVK9BhvxzQ 2 0 2286089 0 1.4gb 1.4gb
green open fs-winlog-2023.07.04 4vVceo8sQYq86z8BM0EyOw 2 0 15980331 0 7.1gb 7.1gb
green open fs-winlog-2023.07.05 fi6AX9lSSFqsAUyzKnqepQ 2 0 2616541 0 1.7gb 1.7gb
green open fs-winlog-2023.07.06 OEIJrk_zSPaCQ6Zp-IweTQ 2 0 2809790 0 1.8gb 1.8gb
green open fs-winlog-2023.07.07 rAHZR7z-Qpi9E6QlkuqD_A 2 0 1945062 0 1.1gb 1.1gb
green open fs-winlog-2023.07.08 ehtf-iToSn-PIylETTEMOg 2 0 717210 0 382.5mb 382.5mb
green open fs-winlog-2023.07.09 w6ExOLyySMe5OIimdb5f5A 2 0 573279 0 347.6mb 347.6mb
green open shrink-cd64-my-cucm-cdr-sec-2023.05 w0lt_7hEQ7Cki_XPbSH5tQ 1 0 368006 0 857.4mb 857.4mb
green open fs-winlog-2023.07.10 3He8zgPYRryU3iBzJO9lqg 2 0 8627163 0 4.8gb 4.8gb
green open fs-winlog-2023.07.11 _FPw5Z6FR26Jqg0PCdmUGQ 2 0 21089525 0 9.4gb 9.4gb
green open fs-winlog-2023.07.12 e0_-SBV9TDugYw5i066pLQ 2 0 4446586 0 2.5gb 2.5gb
green open fs-winlog-2023.07.13 pViI74ZlT8yUDeKsf0AiWw 2 0 2715123 0 1.8gb 1.8gb
green open fs-winlog-2023.07.14 6AXNC_PaRpiCRLSIVFx_0A 2 0 2339736 0 1.5gb 1.5gb
green open fs-winlog-2023.07.15 8iLJ_MMXTWGWFiQuKGwxQA 2 0 583959 0 315.6mb 315.6mb
green open fs-winlog-2023.07.16 h9ucnMhDQJ-2dkzyXDyXfg 2 0 679480 0 379.8mb 379.8mb
green open fs-winlog-2023.07.17 7SZv0BqzQsCMG0Yq0aMFjA 2 0 3297234 0 2gb 2gb
green open fs-winlog-2023.07.18 kfkKyBpzTHaZejwWGp52Ag 2 0 12994545 0 6.1gb 6.1gb
green open fs-winlog-2023.07.19 LbXugOw3QRiYY6ITthpAKw 2 0 3054874 0 1.9gb 1.9gb
green open block-ngfw-2023 WBOgZBqFTYyp05dvgUx4vQ 2 1 14211091 0 1.9gb 993.3mb
green open pexip-syslog-2023.10 CKQv2pesTkOv_1QiFBdQLg 2 1 10283532 0 10.9gb 5.4gb
green open testmap oSrvPxE3Qcugvb7O1-DkgA 1 1 2 0 9.9kb 4.9kb
green open pexip-syslog-2023.11 cij0ZJ6EQRyJWeqo2zmqmA 2 1 2129843 0 2.9gb 1.5gb
green open winlogbeat-srv-login-2023.01_1 UmvAV2CnRwSAShoP3NzBGQ 2 0 14359552 0 5.4gb 5.4gb
green open .kibana-observability-ai-assistant-kb-000001 2CnAsQWjRo66q83ffL6FJA 1 1 0 0 496b 248b
green open fs-winlog-2023.07.20 zHnn_Kc8TZiyZKmGhcQZDA 2 0 7167337 0 4.2gb 4.2gb
green open fs-winlog-2023.07.21 jC13y4PsRF6HRBjwwc-s7g 2 0 2197731 0 1.4gb 1.4gb
green open fs-winlog-2023.07.22 ZzDlWWB9Rn2ynxEjBLC0fw 2 0 750808 0 416.7mb 416.7mb
green open fs-winlog-2023.07.23 wO7IisRJRPOYCfV3P7uYAQ 2 0 569847 0 362.1mb 362.1mb
green open fs-winlog-2023.07.24 5r7XJTCNQLadQ4EpJIdy7Q 2 0 4753363 0 2.8gb 2.8gb
green open fs-winlog-2023.07.25 3psQz56MT5GtcSXR60tARQ 2 0 2843587 0 1.8gb 1.8gb
green open fs-winlog-2023.07.26 IUvzfz3gRcKIY_UDCvcZeA 2 0 4361718 0 2.3gb 2.3gb
green open fs-winlog-2023.07.27 fJyEefdfSmOXGm0VDq1IPg 2 0 12915920 0 6.3gb 6.3gb
green open fs-winlog-2023.07.28 -p5Q5KVDTrqZnNbOj1tTvA 2 0 1933960 0 1.1gb 1.1gb
green open fs-winlog-2023.07.29 Mje8DH0LRESagNzqFghZDA 2 0 828798 0 450.1mb 450.1mb
green open exchange-v-exch-fe01-2023.10 0IW6UXIMSgqqd1IU9mAnzQ 2 1 6560050 0 20.4gb 10.2gb
green open fs-winlog-2023.07.30 kLQHnWR0SIWbd2NR8gBCzg 2 0 507150 0 323.4mb 323.4mb
green open fs-winlog-2023.07.31 oAc8yx-7TGmQRzviqTQ1Cw 2 0 3762576 0 2.4gb 2.4gb
green open exchange-v-exch-fe01-2023.11 GV1Q-B3TSZu55PWgiAdRKA 2 1 1649285 0 5.7gb 2.9gb
green open employeeremotework-2023.08.04 1FrGH6eiRUGaTm-IgP-tzA 2 1 2619 0 1.1mb 583.2kb
green open aws-s3-sts-2023.11.04 ND3rJHN9T4ysGjzbCT4Q9A 1 1 202436 0 895.6mb 452.1mb
green open aws-s3-sts-2023.11.05 IXIuW7qDQYuAw3hRHxpfcw 1 1 188813 0 827.9mb 419.5mb
green open aws-s3-sts-2023.11.06 1XTQMHqoQHugWSqlIopWkA 1 1 178986 0 788.7mb 394.5mb
green open aws-s3-sts-2023.11.07 wQla-XYyTV-ssiry5IG2UA 1 1 71447 0 315.2mb 162.3mb
green open winlogbeat-srv-event_code-2023.02 pQrgDjGlRA2Gm_7F0wmdeg 2 1 604 0 1.5mb 801kb
green open test-map-user2 IbniWyeqTPua5SrOgp1wHg 1 1 1 0 7.7kb 3.8kb
green open winlogbeat-srv-event_code-2023.04 Am5Xc3lyR2WYr5bge61q-w 2 1 394386 0 690.9mb 345.4mb
green open winlogbeat-srv-event_code-2023.05 XQ1wvDhHRL6rtVnSb8pPZw 2 1 600571 0 971.1mb 489mb
green open winlogbeat-srv-event_code-2023.06 inxl3hjpSA2Xf723gD2d3w 2 1 1045712 0 1.3gb 696.8mb
green open winlogbeat-srv-event_code-2023.07 LJH2cnJkRyWQjZyIzoT34w 2 1 587429 0 939.1mb 469.5mb
green open winlogbeat-srv-event_code-2023.08 SUEFz17dTwSz0GSomQdFew 2 1 562722 0 990.8mb 492.8mb
green open winlogbeat-srv-event_code-2023.09 ItcMOuGDQ3yJpYiBc2Fqdg 2 1 564188 0 1012.9mb 498.2mb
green open .lists-default-000001 HJpZViDcRS-Lq_aihNObeQ 1 1 0 0 496b 248b
green open my-ngfw-syslog-2023.10.08 qJT-p5cUTxmC4R_sWT3qjQ 2 0 103605 0 17.8mb 17.8mb
green open my-ngfw-syslog-2023.10.09 zuBIFk4lSt6kdwpSGSVLKg 2 0 106654 0 19.9mb 19.9mb
green open employeeremotework-2023.08.11 7NGSPTjGSPm7LjaZTcmYuA 2 1 2592 0 1mb 564.5kb
green open .kibana-observability-ai-assistant-conversations-000001 wDzxg-vVTnGGI2K9BH11Hw 1 1 0 0 496b 248b
green open employeeremotework-2023.08.18 OyeGcGfWQTiIuU786dUtdQ 2 1 2609 0 1.1mb 564kb
```
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [November 7, 2023, 9:39am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/7 "2023-11-07T09:39:10Z")
...
```auto
green open aws-s3-ecs-2023.11.04 ar6ygbA9TXi9XIWW5E-CaA 1 1 20355 0 32.6mb 17.5mb
green open aws-s3-ecs-2023.11.05 0mfPkMiISBKpts4gINoODA 1 1 20160 0 31mb 14.6mb
green open winlogbeat-srv-event_code-2023.10 kPJm_zGTQ3uy9Ku9B_8Z7g 2 1 686675 0 1.1gb 589mb
green open winlogbeat-srv-event_code-2023.11 prajcXnwQ2it7WN5R7pruw 2 1 220220 0 362.2mb 171.2mb
green open aws-s3-ecs-2023.11.06 AkRAVE5WQr63rX7NIV9a9w 1 1 20079 0 31.6mb 14.6mb
green open aws-s3-ecs-2023.11.07 acZY8IgzRYShjWOv7b0OqQ 1 1 7733 0 12.5mb 6.4mb
green open filebeat-erp-aud-v-sap-perpn10-2023.10.01 ITtxvAmwQxKx1oy4l8PsnA 2 1 12061750 0 4.5gb 2.2gb
green open filebeat-erp-aud-v-sap-perpn10-2023.10.02 BTFS8Lk-SnG9SDVRY4ixqQ 2 1 11520904 0 4.7gb 2.3gb
green open filebeat-erp-aud-v-sap-perpn10-2023.10.03 ho4-C60xSnqlh6B2ANmvQw 2 1 8419259 0 3.4gb 1.7gb
green open filebeat-erp-aud-v-sap-perpn10-2023.10.04 xzADvAmiQZGiuu6tfsiGsA 2 1 6190804 0 2.3gb 1.1gb
green open filebeat-erp-aud-v-sap-perpn10-2023.10.05 y_KZ5W-_QPaOiN5ZzHtaOw 2 1 6155569 0 2.2gb 1.1gb
green open filebeat-erp-aud-v-sap-perpn10-2023.10.06 ML5Tq5W0QtqEGCqS5qrevA 2 1 6367640 0 2.2gb 1.1gb
green open filebeat-erp-aud-v-sap-perpn10-2023.10.07 YAZ5g9MsTS2jxgJY0lEoaw 2 1 2029179 0 703.8mb 351.9mb
green open filebeat-erp-aud-v-sap-perpn10-2023.10.08 -CA7fe26QJOXVb_vbh1c1Q 2 1 1362335 0 558.3mb 279.1mb
green open oss-inventory-phone-2023.09 SQQ9zkVZSrykqKMft3Jt2w 4 1 1370 0 2.7mb 1.3mb
green open filebeat-erp-aud-v-sap-perpn10-2023.10.09 eoMnFtkhSMqlS0dwjpiRFw 2 1 2411336 0 1.2gb 639.6mb
green open my-ngfw-syslog-2023.10.19 i5h1zWcTT9md3K86a4gAGg 2 0 105253 0 20mb 20mb
green open shrink-loni-my-cucm-cdr-sec-2023.07 hb2Hxm_xQVi1GwgOXSn6hA 1 0 523127 0 1.2gb 1.2gb
green open my-ngfw-syslog-2023.10.17 GqMNOY3mSom-bv-BXvVh7Q 2 0 98419 0 16.1mb 16.1mb
green open my-ngfw-syslog-2023.10.18 6rhh9p8-QMqHLp10LNzc5w 2 0 106915 0 17.9mb 17.9mb
green open my-ngfw-syslog-2023.10.11 zgr6mO5LQcuZ_JNM6AtJzQ 2 0 105800 0 20mb 20mb
green open my-ngfw-syslog-2023.10.12 pwEn5T_6R5aE14UZtMyGyg 2 0 107507 0 19.5mb 19.5mb
green open my-ngfw-syslog-2023.10.10 CPQKGnmjRE21Z_PPT3VnGQ 2 0 105929 0 21.4mb 21.4mb
green open my-ngfw-syslog-2023.10.15 JEj55RwpRkKBsBTh9INBzA 2 0 103652 0 18.9mb 18.9mb
green open my-ngfw-syslog-2023.10.16 vaEhmWSIS_Sgs4OpsXteQA 2 0 104887 0 19mb 19mb
green open my-ngfw-syslog-2023.10.13 5a27vO62ReWiboB-o7Qogw 2 0 108258 0 20.7mb 20.7mb
green open my-ngfw-syslog-2023.10.14 d9t4pc8DSzOBTFRqO9yZNA 2 0 104016 0 17.1mb 17.1mb
green open azurevpn-olwayson-2023.01 cvaQ1K-hTbyg-1lPSz9jnQ 2 1 9875272 0 10.8gb 5.4gb
green open employeeremotework-2023.04.07 IG4gL736QsGjNKYPFQRPug 2 1 2519 0 1mb 535.4kb
green open azurevpn-olwayson-2023.02 mbQ0z6cJRTmGUVxpj6dNJg 2 1 4388876 0 4.5gb 2.2gb
green open azurevpn-olwayson-2023.04 4N-EW2TzSNqOttd7RPoWNg 2 1 3016559 0 3.1gb 1.5gb
green open azurevpn-olwayson-2023.05 VCju1GrTQb2r9NCbd05XGQ 2 1 3107699 0 3.2gb 1.6gb
green open azurevpn-olwayson-2023.06 E_hQIgPURoKgS29IOxRH6w 2 1 133733960 0 160.9gb 80.4gb
green open azurevpn-olwayson-2023.07 JPHxUoQpTiOGRkDjvAOU8g 2 1 80653612 0 97.5gb 48.7gb
green open azurevpn-olwayson-2023.08 UJdcv47TR7S411rYl33q5w 2 1 706 0 1mb 521.4kb
green open azurevpn-olwayson-2023.09 ILDvzzxQRc-xc2ugBLLyHQ 2 1 883 0 1.1mb 490.6kb
green open oss-inventory-phone-2023.10 AHdl_CyYSwyt94YnS7TKVA 4 1 4925 0 9mb 4.4mb
green open oss-inventory-phone-2023.11 59dpkdUWSh-9aCCPhxwaTA 4 1 1128 0 2.2mb 1.2mb
green open azurevpn-olwayson-2023.03 h7lzlMn2RzuSRgnzOckolw 2 1 3877767 0 3.9gb 1.9gb
green open oss-voice-gw-2023.09 TbKxyis2SHGz5ipD3l5ucw 4 1 1034812 0 550.9mb 278.5mb
green open tst-all-2023.11 RUd7_p4LQSSBq_X4871j4w 2 1 12 0 194.8kb 97.4kb
green open employeeremotework-2023.08.25 xH-YmA8rSR-JHtryFOG9Kw 2 1 2595 0 1.3mb 732.6kb
green open my-ngfw-syslog-2023.10.22 WSTei8eKQl2Vi6JgmoYX7w 2 0 103559 0 28.1mb 28.1mb
green open my-ngfw-syslog-2023.10.23 JJq28ofoSEOIYaiO_bm7CA 2 0 103864 0 32.1mb 32.1mb
green open my-ngfw-syslog-2023.10.20 Mg7Ve3oZTqiKbBHZYzOQ3A 2 0 106447 0 18.2mb 18.2mb
green open metricbeat-dc-2023.11.01 i81W-104RJS1AItFpmv6tA 2 0 6786100 0 1.9gb 1.9gb
green open metricbeat-dc-2023.11.02 -9_TcpxeQd2RdPaXDxxLTw 2 0 6712079 0 1.9gb 1.9gb
green open metricbeat-dc-2023.11.03 0JE4n7hMT6mBfbMp0TLoYQ 2 0 6665412 0 1.9gb 1.9gb
green open metricbeat-dc-2023.11.04 zC1MACkeT6qKueg25Fg4Fg 2 0 6668156 0 1.9gb 1.9gb
green open metricbeat-dc-2023.11.05 LKSpyzd5RdS8gRneB_Yw8g 2 0 6661903 0 1.9gb 1.9gb
green open oss-voice-gw-2023.10 oyAWNsIKS8y7ZdZB3tFadA 4 1 202063 0 154.9mb 76.9mb
green open winlogbeat-spdev-2023.11.04 yvsQKWjGRV6ULECe0WPDjA 2 0 143 0 890.5kb 890.5kb
green open winlogbeat-spdev-2023.11.05 e-zQFU62Q5CJ-38SuM8AFA 2 0 120 0 676.3kb 676.3kb
green open winlogbeat-spdev-2023.11.06 UhX7DRLKQlqOz_REASUbrg 2 0 112 0 587.3kb 587.3kb
green open winlogbeat-spdev-2023.11.07 6XPzAR8rTpWj8AIMau48MA 2 0 47 0 552.5kb 552.5kb
green open oss-voice-gw-2023.11 XZIitMNVRiK2nUsAO45hKw 4 1 73751 0 38.1mb 17.6mb
green open employeeremotework-2023.04.14 5UoRhYGISD2tEXoIQgnqfA 2 1 2523 0 1mb 531.8kb
green open metricbeat-dc-2023.11.07 Jo4wCsqbSSeAKFb7CJ2xXQ 2 0 2695373 0 826.2mb 826.2mb
green open metricbeat-dc-2023.11.06 hRrbwTQgQ-a3eL9WT5ROpg 2 0 6710490 0 1.9gb 1.9gb
green open ftp-logs-2023.02 zBO973Q0RGCPRpKyrvjCZg 2 1 123303422 0 36.4gb 18.1gb
green open ftp-logs-2023.04 ri4w26cxQvao0sdii_NUXQ 2 1 132421843 0 37gb 18.5gb
green open ftp-logs-2023.05 yhi5jAq8SXe5P53Jeic8BQ 2 1 131783772 0 36.8gb 18.4gb
green open ftp-logs-2023.06 PkxH6p0uSB2Ju9xBEK2CrA 2 0 122488389 0 17.9gb 17.9gb
green open ftp-logs-2023.07 G_NGuf1FTPe3Xpf8jOjegA 2 0 91483243 0 13.2gb 13.2gb
green open ftp-logs-2023.08 jv78KrdVTaa75Uq7ZvWd3A 2 0 98681766 0 14.6gb 14.6gb
green open ftp-logs-2023.09 wJjIV-mRQOeMWb73qRpjAA 2 0 95947998 0 14.3gb 14.3gb
green open ftp-logs-2023.03 9d_IaluVQGaJ4LFlyAWy9A 2 1 160840753 0 46.3gb 23.1gb
green open ftp-logs-2023.01 Gyo__AolR8eIro7TsTavuw 2 1 91001298 0 30.3gb 15.2gb
green open grids-cuic-2023 jDALmKdYTO2GzTABZB7-Gg 2 1 1 0 24.7kb 12.3kb
green open azurevpn-olwayson-2023.11 T1PXp7v3TtqeluJekLo4Qg 2 1 176 0 491.4kb 238.9kb
green open azurevpn-olwayson-2023.10 -VTpOfc2RpmL9NMbZqf7Gw 2 1 554 0 966.1kb 486.6kb
green open winlogbeat-srv-sdc-2023.11.04 6wZ_UK4EQ0arv89pQjxQMw 2 0 33567276 0 31.3gb 31.3gb
green open winlogbeat-srv-sdc-2023.11.06 3egl3-hpTCWD72c2KfFRPQ 2 0 51339047 0 46gb 46gb
green open winlogbeat-srv-sdc-2023.11.05 R2KE3agcRjalgbOZI-64ww 2 0 36555891 0 34.9gb 34.9gb
green open winlogbeat-srv-sdc-2023.11.07 lGMazAySSKSRRfdo-fa2uA 2 0 20264326 0 18.5gb 18.5gb
green open my-ngfw-syslog-2023.10.28 JzVLAogoSBO5Bb9RBmmaGw 2 0 104531 0 18.5mb 18.5mb
green open my-ngfw-syslog-2023.10.29 ag26N8qoTg-YvX94KbaspQ 2 0 103542 0 19.5mb 19.5mb
green open my-ngfw-syslog-2023.10.21 DoeU-X8iT7u9QWEZNFjhGw 2 0 104139 0 17.4mb 17.4mb
green open .items-default-000001 -2fc6ljVSwSe0fd58aIy2w 1 1 0 0 496b 248b
green open api 8ckvvQTnQRCYG3nR-bIN2Q 1 1 1 0 9.4kb 4.7kb
green open my-ngfw-syslog-2023.10.26 oMP_u2fCRPaR5bbyNLn-vQ 2 0 77881 0 13mb 13mb
green open my-ngfw-syslog-2023.10.27 7qIutnrCQAWYB8oOEiXygg 2 0 107140 0 20.9mb 20.9mb
green open my-ngfw-syslog-2023.10.24 CTYhe5SCRmCT07JaV5vZ0A 2 0 85841 0 18.9mb 18.9mb
green open my-ngfw-syslog-2023.10.25 _VQTPpqVQXmc0nNczeHzpA 2 0 53528 0 10.6mb 10.6mb
green open employeeremotework-2023.04.21 DbwckRB5TqC5IqnNhgxW5w 2 1 2522 0 1mb 553.1kb
green open .internal.alerts-observability.uptime.alerts-default-000001 QJFHdDFhTlSy2T_gzUyNqg 1 1 0 0 496b 248b
green open my-ngfw-syslog-2023.10.30 cD1lktU2QK6O-rn0o_M0qQ 2 0 106670 0 28.7mb 28.7mb
green open ftp-logs-2023.10 FvhZ0t3ERjq5FQQnuVeb6g 2 0 99045577 0 14.7gb 14.7gb
green open ftp-logs-2023.11 leNYmQ8lQ1WApoyfb0X29g 2 0 19650539 0 3.1gb 3.1gb
green open my-ngfw-syslog-2023.10.31 WIpXgdhaRKGiNWnNThlAAw 2 0 105417 0 28.2mb 28.2mb
green open employeeremotework-2023.04.28 6cyhax9KTjaSzMAqS7GAjA 2 1 2534 0 1mb 522.3kb
green open cssm-syslog-2023.10 5YdoWnkqTRCnKbPYKYjXUg 2 1 61762902 0 17.1gb 8.5gb
green open cssm-syslog-2023.11 JkUHY41OR-qmuLXAgXFfXw 2 1 10950286 0 3.4gb 1.6gb
green open shrink-ttjp-my-cucm-cdr-sec-2023.08 EItNJvb6TaCDxHCYzSKhSQ 1 0 476319 0 1.1gb 1.1gb
green open winlogbeat-srv-user-workstation-geo-2023.01 Wb4wRNtiQ5GQosoTSwlZmA 2 1 4508781 966337 3.6gb 1.8gb
green open winlogbeat-srv-user-workstation-geo-2023.02 Wh4vBxx7Rjuw8tYzZ8aYKg 2 1 10826602 3505121 8.3gb 4.2gb
green open shrink-wxde-my-cucm-cdr-sec-2023.06 C_f7fuTySUib550-46WxrQ 1 0 758569 0 1.6gb 1.6gb
green open winlogbeat-srv-user-workstation-geo-2023.05 urgM9NmbTtGTCLPncwFo1w 2 1 15215782 1930536 11.7gb 5.8gb
green open winlogbeat-srv-user-workstation-geo-2023.04 fQul7HxrT52J7hG3TLFQrw 2 1 7008970 1640362 6.9gb 3.4gb
green open winlogbeat-srv-user-workstation-geo-2023.07 adCedVQsTXaATPgQ2zPsSA 2 1 15189071 2646031 11.7gb 5.9gb
green open winlogbeat-srv-user-workstation-geo-2023.06 zuzAUK4dTaaPxFAAF-puFA 2 1 15560002 3239960 12.4gb 6.2gb
green open winlogbeat-srv-user-workstation-geo-2023.09 CvgI3Ur7S_mRETR_FBs-5g 2 1 4186881 23446 3.1gb 1.6gb
green open winlogbeat-srv-user-workstation-geo-2023.08 RkpYW_MkQwarw57ybaBP1A 2 1 10702026 1386321 8.3gb 4.1gb
green open metrics-index_pattern_placeholder mOSudl3SS_apWQXC-6UWZg 1 1 0 0 496b 248b
green open itsm-users CLYLzUrCQH-sNiUf2FJrTA 1 1 103300 38253 201.2mb 100.6mb
green open winlogbeat-srv-user-workstation-geo-2023.10 5AJ49sidSgiJIZ5IS1TwOw 2 1 2354593 27863 1.8gb 950.5mb
green open o365-token E8oO1QgMTqivcdMwfh3cKg 2 1 1 0 43.5kb 21.7kb
green open db-archive-device XLnUzP-HSOSBhxI7-sQnnA 1 1 14284 0 22.2mb 11.1mb
green open o365-issue-update uDRfrLcYTdew4GsWP-hk4g 2 1 1718 6943 75.2mb 37.5mb
green open fs-winlog-2023.08.01 jx_S6DZBRSyL5WVitTqJZQ 2 0 16227406 0 6.9gb 6.9gb
green open fs-winlog-2023.08.02 kmLloUAmTsq7rYgfUTE-MQ 2 0 4058986 0 2.5gb 2.5gb
green open fs-winlog-2023.08.03 gR0MKBTDR-WuFJaofHPknQ 2 0 3475869 0 2.2gb 2.2gb
green open fs-winlog-2023.08.04 n2CgIvWBSLSrklBwhoxjrA 2 0 2418045 0 1.5gb 1.5gb
green open fs-winlog-2023.08.05 7W_MjcpCRBu4cWwNvUUb0Q 2 0 910607 0 511.6mb 511.6mb
green open fs-winlog-2023.08.06 9_YwYELlTNyLxmySVkz5kA 2 0 681534 0 425.9mb 425.9mb
green open fs-winlog-2023.08.07 E2qM_NwaSHOlR7WWut8Bxw 2 0 2630083 0 1.8gb 1.8gb
green open .monitoring-es-7-2023.11.01 Yu0wK4ksTz-e_O1sSSxC0Q 1 1 6687937 76563 5.9gb 2.9gb
green open .monitoring-es-7-2023.11.02 B37E4K2jSRykFmYS9imTqQ 1 1 6174145 43570 5.7gb 2.8gb
green open .monitoring-es-7-2023.11.03 grA4gDT6SmC7kMJ-VOYIzA 1 1 6188413 100008 5.7gb 2.8gb
green open .monitoring-es-7-2023.11.04 0lCwOcsiRIGh_QxV6Atl2g 1 1 6095183 63388 5.6gb 2.8gb
green open .monitoring-es-7-2023.11.05 EEbPADF-RpmnXZzUiETZeA 1 1 6078128 81147 5.6gb 2.8gb
green open .monitoring-es-7-2023.11.06 Q5uGczisQ9i-RM8k8hLzrg 1 1 6061194 59410 5.6gb 2.8gb
green open .monitoring-es-7-2023.11.07 wq3D9cgkSPK21eXUEar_iw 1 1 2608945 35636 2.2gb 1.1gb
green open yasno-cuic-2023 YIofWlPmTFWg_2OgQqiMlg 2 1 13 0 224.8kb 108.2kb
green open filebeat-8.3.3 kVQllFDgRnylx0xdcT6Hhw 1 1 5 0 143.6kb 71.8kb
green open fs-winlog-2023.08.08 git3GasdRIWhwQXEMtBRLw 2 0 8619019 0 4gb 4gb
green open fs-winlog-2023.08.09 ROpAsYfYT-qSlSClZLhQlA 2 0 2776251 0 1.8gb 1.8gb
green open my-auditlog-ucsyslog-2023.10 k5-kuAJMT7CiR8nsTfEzTA 2 1 596397 0 536.5mb 267.9mb
green open my-auditlog-ucsyslog-2023.11 HI3-0r1_S5alRWvEamoNMw 2 1 157757 0 133.7mb 64.9mb
green open filebeat-sap-v-sap-wcr01-2023.11 WKboxoeaQCuoC0Q1inw3-Q 2 0 14992333 0 4.7gb 4.7gb
green open filebeat-sap-v-sap-wcr01-2023.10 4fB3jEDdQGOwy5Y3gDMiig 2 0 61186731 0 18.9gb 18.9gb
green open fs-winlog-2023.08.10 vZneAHFAQbagHTATbcW2XA 2 0 3828051 0 2.2gb 2.2gb
green open fs-winlog-2023.08.11 nS0Pe9bSRPi8JuE0ydr9kw 2 0 2753589 0 1.8gb 1.8gb
green open fs-winlog-2023.08.12 q7wK9YGGT3K-mOdew3DdHg 2 0 17625972 0 7gb 7gb
green open fs-winlog-2023.08.13 L8K0vjFzS_WoJrpCep3Uow 2 0 6227321 0 2.2gb 2.2gb
green open fs-winlog-2023.08.14 Jor3WOGzSp2P4651ogK29w 2 0 17145660 0 8.4gb 8.4gb
green open fs-winlog-2023.08.15 Xe5Dy46ISuSvxIGeufa5dg 2 0 8476624 0 4.1gb 4.1gb
green open fs-winlog-2023.08.16 OsSvvFuBQla-0dB-C-l_fw 2 0 23608215 0 10.3gb 10.3gb
green open fs-winlog-2023.08.17 Eka-_0tYQUSUwA4dDzODPw 2 0 10538389 0 5.1gb 5.1gb
green open fs-winlog-2023.08.18 f8Oe_OjqRnClBjH2FpRpOQ 2 0 2417176 0 1.5gb 1.5gb
green open userdevicevpn-total vB2s6gbgRc2sxMBdB0Ps0A 1 1 40983 0 6.9mb 3.4mb
green open fs-winlog-2023.08.19 -yAyWOTSSmuCFvrYV7QnfQ 2 0 721244 0 376.9mb 376.9mb
green open mfa-status-authenticationmethods-2023.11.04 vBQBeoOURv27X2vaYLfakw 2 1 812924 0 421.8mb 216.4mb
green open mfa-status-authenticationmethods-2023.11.05 EdaAyYHqTHeXUtrrfii9qA 2 1 613326 0 333.4mb 165.4mb
green open mfa-status-authenticationmethods-2023.11.06 OyeMBGN0TOenFsH3_Z4quA 2 1 642373 0 354.3mb 170.7mb
green open mfa-status-authenticationmethods-2023.11.07 yhZwFxHxQ6iSNHXEiUZQFg 2 1 330856 0 191.9mb 93.4mb
green open fs-winlog-2023.08.20 RwNGiYVcQmaeIQjiiS956A 2 0 506116 0 302.6mb 302.6mb
green open fs-winlog-2023.08.21 1_aMCU1-SN-P_QCaab-Y5A 2 0 1685193 0 1gb 1gb
green open fs-winlog-2023.08.22 IGr69DzKTFqn3VOcT-rB_w 2 0 7530813 0 3.1gb 3.1gb
green open fs-winlog-2023.08.23 GkQlaBnjTJKspLm6wqv7jg 2 0 3998540 0 2.3gb 2.3gb
green open fs-winlog-2023.08.24 LEJ4MhWZTR-HrxgpoUZQfg 2 0 2603495 0 1.5gb 1.5gb
green open fs-winlog-2023.08.25 TeqKdGXAQvSmxB61xQQM7g 2 0 3501810 0 2.1gb 2.1gb
green open fs-winlog-2023.08.26 tLf11IyXQtC5ABkhdVuDgA 2 0 960301 0 604.9mb 604.9mb
green open fs-winlog-2023.08.27 Yu0ZoH4PSSap9X_NdJkoMA 2 0 612132 0 397.6mb 397.6mb
green open fs-winlog-2023.08.28 oNexWX8fSA6g6EIhAG3iBg 2 0 2727051 0 1.8gb 1.8gb
green open fs-winlog-2023.08.29 opZ8rRUISn-XpcaINElURw 2 0 8977014 0 4.2gb 4.2gb
green open cvp-gcc-app-2023.10 BteAHhURR5WfdLyMaKk0TQ 2 0 1301 0 1006kb 1006kb
green open fs-winlog-2023.08.30 fuM3HS5sSDqdOZhRRPHnww 2 0 3034054 0 2gb 2gb
green open fs-winlog-2023.08.31 0kjeGVbLTNG9WF-QF9RrFg 2 0 2873781 0 1.9gb 1.9gb
green open employeeremotework-2023.09.01 TE2I_w6TQd-DYgN__7VIpQ 2 1 2594 0 1.4mb 668.9kb
green open winlogbeat-srv-userlogin-2023.06 sePA0HJfS5mAgeUVOPplbw 2 0 1080753 0 660.3mb 660.3mb
green open winlogbeat-srv-userlogin-2023.07 bIIj-LKvSKSZOR2T0Bqasg 2 0 6669758 0 3.6gb 3.6gb
green open winlogbeat-srv-userlogin-2023.08 rkRrowzyStySy8fm6fPdXw 2 0 6268797 0 3.5gb 3.5gb
green open winlogbeat-srv-userlogin-2023.09 -uu9m1ypR-6FHx0RTJtVZQ 2 0 6395897 0 3.6gb 3.6gb
green open employeeremotework-2023.09.08 GaABaOkZQguc_FxkVIJUSA 2 1 2613 0 1.5mb 784.1kb
green open my-grok-ucsyslog-2023.10 IyQzM0QTS0W9AdZfD7Ixwg 2 1 35049628 0 9.6gb 4.8gb
green open my-grok-ucsyslog-2023.11 gGTEQHqvRyWBHrfngKMl5Q 2 1 6574908 0 1.8gb 990.8mb
green open winlogbeat-srv-user-workstation-2023.03-new V1HmxJXGSMSyaLoJhY8uYA 2 1 13620986 0 8.4gb 4.2gb
green open my-ngfw-syslog-2023.11.01 qZDoY_YCSTu3sWX8Wd4Ppg 2 0 106963 0 28.3mb 28.3mb
green open my-ngfw-syslog-2023.11.02 xgWAPctmR0WTpx4qItRNFg 2 0 106509 0 19.2mb 19.2mb
green open my-ngfw-syslog-2023.11.03 ZiyVfyO2SYWDr1-yBFXayg 2 0 106438 0 19.5mb 19.5mb
green open my-ngfw-syslog-2023.11.04 WJku6V9eRkOnjm4gvbszRQ 2 0 104037 0 16.7mb 16.7mb
green open my-ngfw-syslog-2023.11.05 K9Dz4MvOSwSPLkex0fpuRA 2 0 102875 0 18.2mb 18.2mb
green open my-ngfw-syslog-2023.11.06 wJx7ex4OQd6PmXyPeSGKNQ 2 0 106381 0 20mb 20mb
green open my-ngfw-syslog-2023.11.07 CZGQ6h3yRA2shs7Ug32unA 2 0 42550 0 10.5mb 10.5mb
green open filebeat-8.3.3-cisco-ios-2023.01 vKbSsoruQzeGj_1EifZtyA 2 0 3060 0 3.2mb 3.2mb
green open filebeat-8.3.3-cisco-ios-2023.02 2DSUpwLNRXiR_32liFzaoQ 2 0 519 0 734.6kb 734.6kb
green open filebeat-8.3.3-cisco-ios-2023.03 EXr4r-oZQHGZfCXLO9bynw 2 0 31036 0 37mb 37mb
green open filebeat-8.3.3-cisco-ios-2023.04 GuX3hsEtTLi3N6hUjx8YpA 2 0 2452 0 3.6mb 3.6mb
```
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [November 7, 2023, 9:39am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/8 "2023-11-07T09:39:37Z")
...
```auto
green open .monitoring-kibana-7-2023.11.01 jIni6cg2QbKdOrdVD69Ctg 1 1 34558 0 19.3mb 11.2mb
green open filebeat-sap-v-sap-wdisp1-2023.10 cPMJR-9zQC2WGyn_zM35Vg 2 0 194603510 0 64.1gb 64.1gb
green open filebeat-sap-v-sap-wdisp1-2023.11 C30qNRMmTyK7D0wynmidyQ 2 0 48885705 0 16.2gb 16.2gb
green open vpn-device-2023 phUSkaS6QTeP98yJUsqdOw 2 1 330039 0 198.3mb 99.1mb
green open .monitoring-kibana-7-2023.11.05 BXBCt7ifR7uK-di1RIUAMA 1 1 34560 0 21mb 10.5mb
green open .monitoring-kibana-7-2023.11.06 rdDI-rJ7RAKwsBqc98DQpA 1 1 34560 0 18.6mb 9.2mb
green open .monitoring-kibana-7-2023.11.07 eOh8rbx0SdSLGggwA7jcpQ 1 1 13750 0 11.2mb 5.6mb
green open .monitoring-kibana-7-2023.11.02 xp6gjl2vQYOKdu_eOF7LUw 1 1 34558 0 18.8mb 9.4mb
green open .monitoring-kibana-7-2023.11.03 HxpzByziRvWgxpjL6j7ITQ 1 1 34558 0 18.6mb 9.2mb
green open .monitoring-kibana-7-2023.11.04 u0PZr99vSHWjVa-8iFDhqA 1 1 34560 0 16.6mb 7.4mb
green open filebeat-8.3.3-cisco-ios-2023.06 CvCMb_K3TXuO7UEjdNup2w 2 0 1446 0 2.3mb 2.3mb
green open winlogbeat-srv-userlogin-2023.11 oNvl3RrNQRSbbsUtsV71FQ 2 0 1464502 0 1gb 1gb
green open winlogbeat-srv-userlogin-2023.10 g_WJMDW4SEaCoyNA5zE7CA 2 0 6763032 0 3.7gb 3.7gb
green open employeeremotework-2023.09.15 uDHXIpO4TcG-bAZ3I8zqUQ 2 1 2610 0 1.4mb 772.7kb
green open winlogbeat-srv-user-workstation-geo-2023.03-new 1BCtTemJSkaKdeHlMROzKQ 2 1 13528654 0 8.6gb 4.3gb
green open .fleet-file-data-endpoint-000001 sDU0a5sQQkGTyexjGZJn6w 1 1 0 0 496b 248b
green open filebeat-8.3.3-cisco-ios-2023.10 r49k9hgXRDWBoxnuvfX6cw 2 0 42222291 0 17gb 17gb
green open filebeat-8.3.3-cisco-ios-2023.11 1vFt-90SRK6RhRlM3iaSRA 2 0 12622468 0 5.2gb 5.2gb
green open filebeat-8.3.3-cisco-ios-2023.12 g6Jlb6TGRFqJFKr6b3S_Vg 2 0 2620 0 4mb 4mb
green open metrics-endpoint.metadata_current_default pDSm7ZBTTMGzVFhpJwKv2g 1 1 0 0 496b 248b
green open metricbeat-exchange-2023.11.01 tAgjED85RA2ZijiD1MMkpg 2 0 2111632 0 800.8mb 800.8mb
green open metricbeat-exchange-2023.11.02 sXmJq3xXRGayIwLUnSe3cA 2 0 2116259 0 801.8mb 801.8mb
green open metricbeat-exchange-2023.11.03 JNdpHEWgR8iH1ZAYK73rxw 2 0 2117765 0 804.2mb 804.2mb
green open .fleet-file-data-agent-000001 phbNJ81gTmG5rjLXnzX-kw 1 1 0 0 496b 248b
green open employeeremotework-2023.05.05 4-IyrbTtREivgS_5E24ZIA 2 1 2543 0 1mb 546.2kb
green open metricbeat-exchange-2023.11.04 2uDo93E_SB-dN0VzTatELA 2 0 2130235 0 798.9mb 798.9mb
green open metricbeat-exchange-2023.11.05 fG7Nfo-ERRyfYciCoNgaWQ 2 0 2130129 0 808.2mb 808.2mb
green open rdweb-vpn-device-2023 18qsc73ySDKoRFsiV-LA7w 1 1 16587716 0 8.2gb 4.1gb
green open metricbeat-exchange-2023.11.06 SBc_aDU6QUapUYNDy9lBhg 2 0 2115524 0 802.2mb 802.2mb
green open metricbeat-exchange-2023.11.07 7yt1i86AS8imATUXA_mpKA 2 0 845374 0 330.4mb 330.4mb
green open employeeremotework-2023.09.29 _Yrdvp7zTOqX5-C-tO7X3Q 2 1 2622 0 1.4mb 685.4kb
green open employeeremotework-2023.09.22 qPGfH5gLS-qy8HFeDC3L7w 2 1 2613 0 1.3mb 714.4kb
green open test-map-user lIfwADBlTF6IyBbbyBLFWA 1 1 1 0 9.5kb 4.7kb
green open test-map-index 8XLVQmlpQEyOP8XaBBSjcg 1 1 0 0 496b 248b
green open .internal.alerts-stack.alerts-default-000001 qVLMC3vGRz6TaAF8ZnEiNA 1 1 0 0 496b 248b
green open employeeremotework-2023.05.12 SMpFzIqVRgq9d8q4EohSjg 2 1 2550 0 1mb 560.3kb
green open aws-s3-sso-2023.11.03 gJbWOXxdTn66MWTG8r7qlg 1 1 190 0 1mb 374.9kb
green open aws-s3-sso-2023.11.04 rGpHjoGXRiip93MOpU6SOw 1 1 13 0 143.5kb 71.7kb
green open aws-s3-sso-2023.11.05 47k-kcBDSl6uXqndnFLkYA 1 1 14 0 221.2kb 110.6kb
green open aws-s3-sso-2023.11.06 8YJmA-NQRE-jLndWGrda-Q 1 1 131 0 846.7kb 253kb
green open aws-s3-sso-2023.11.07 fp8s_q3_T4qB9Vq6aHvISg 1 1 20 0 235.9kb 117.9kb
green open employeeremotework-2023.05.19 Q7GrzkeqRwWE9TnqfWWSRg 2 1 2554 0 1mb 534.9kb
green open filebeat-sap-v-sap-webput1-2023.10 7M4GMUP_Rqu_zRNUVXyHvw 2 0 13469953 0 2.8gb 2.8gb
green open winlogbeat-srv-login-2022.12_1 OWaqojpzTGaCYJOe337nqw 2 0 20025 0 7.3mb 7.3mb
green open filebeat-sap-v-sap-webput1-2023.11 JwoR2bKWQ2iIgdCq-uvVKw 2 0 6658515 0 1.4gb 1.4gb
green open my-grok-uccxsyslog-2023.10 sOvv5YnMQqi7PEw9xxWTRw 2 1 1091429 0 575.8mb 289.7mb
green open my-grok-uccxsyslog-2023.11 gVdH5Um3SteIcfW8sX8X8g 2 1 99620 0 96.1mb 42.6mb
green open employeeremotework-2023.01.01 mRbDrwcnTKGoSyQL0V3PNg 2 1 60 0 230.4kb 123.3kb
green open employeeremotework-2023.05.26 ITXMJDdeTEWagT0nRlMEyA 2 1 2559 0 1mb 502.7kb
green open rsyslog-all-2023.11.02 55ri-e2eTgmHK4OZETl7kA 2 1 1599654 0 774.5mb 387.1mb
green open rsyslog-all-2023.11.03 zqlO9Z88SyezQu7tSNLNMQ 2 1 1596077 0 771.2mb 385.1mb
green open rsyslog-all-2023.11.04 N-509ipZQGijoKZEDjptsA 2 1 1597781 0 766.5mb 383.1mb
green open rsyslog-all-2023.11.01 kB8Ue2eJRb-9_CmjwSolyw 2 1 1608573 0 783.6mb 390.9mb
green open rsyslog-all-2023.11.06 SQFPPOZ3STGqCA2PuQfSig 2 1 1598180 0 773.6mb 386.9mb
green open rsyslog-all-2023.11.07 gUhx1tWRSYyIyJ_bd16rbQ 2 1 639982 0 312.8mb 155.9mb
green open rsyslog-all-2023.11.05 mGGkJcDoS1-00MCBKUHgPg 2 1 1584921 0 763.6mb 379.6mb
green open itsm-netwok-device PGP49OULS9GDuMfp9c1d5Q 1 1 6514 9594 18mb 9mb
green open employeeremotework-2023.01.15 doT8zI_6TWy8Vj5hm1HUOQ 2 1 60 0 239.8kb 115.8kb
green open azurevpn-device-2022.12 KLoHAZixT06UBY7OBlzMGQ 2 1 2 0 25.6kb 12.8kb
green open exchange-v-exch-fe04-2023.10 6cPWeRQ4TZu24SV0CkKPhw 2 1 6404849 0 21.1gb 10.5gb
green open exchange-v-exch-fe04-2023.11 ynSX9lNMTZGKiuNICdW2Fg 2 1 1285825 0 4.7gb 2.4gb
green open testmap2 AS5EiX78QC-wfcnWDajEfA 1 1 3 0 8.7kb 4.3kb
green open filebeat-aci-2023.10 a4tm_AN1T0mcsxB-QyNHQA 2 0 1194342 0 487.6mb 487.6mb
green open rdweb-vpn-device-1 iq1gudqlTySh_rkTBHqjlw 1 1 330039 0 201mb 100.5mb
green open filebeat-aci-2023.11 jJFvTQpqQS2l_YDrnPJyzQ 2 0 227009 0 146.8mb 146.8mb
green open rdweb-dev-2023 YwyeHjfuQJuZ-tkAUhFl9Q 2 1 19919292 0 2.5gb 1.2gb
green open testmap3 1MhX2hV7SzuwdTO1sh-yBA 1 1 99 0 29.3kb 14.6kb
green open testmap4 JhhXS9uERV-Nllxn4rzFLQ 1 1 99 0 31.8kb 15.9kb
```
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 7, 2023, 1:46pm UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/9 "2023-11-07T13:46:36Z")
I can see around 500 shards per data node. How much HEAP do you have on data nodes? How much heap do you have on master nodes?
You are using 2 primary shards per index. Why this?
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [November 8, 2023, 10:08am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/10 "2023-11-08T10:08:37Z")
hi, [dadoonet](https://discuss.elastic.co/u/dadoonet)
- config masers 3 pieces
CPU= 2
RAM=8 Gb
HDD=100 Gb
- data node 4 pieces
CPU=12
RAM=64 Gb
HDD= 1 Tb -database
HDD=120 Gb - OS
```auto
● elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─override.conf
Active: active (running) since Mon 2023-10-23 10:05:29 UTC; 2 weeks 1 day ago
Docs: https://www.elastic.co
Main PID: 988 (java)
Tasks: 214 (limit: 77023)
Memory: 57.9G
CPU: 1w 1d 12h 58min 20.143s
CGroup: /system.slice/elasticsearch.service
├─ 988 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcl>
├─1345 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+>
└─1456 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
```
```auto
jvm.options
...
-Xms31g
-Xmx31g
...
```
- kibana node 2 pieces
CPU=4
RAM=8 Gb
HDD= 100 Gb
- ingest node 2 pieces
CPU=4
RAM=8 Gb
HDD= 50 Gb
- logstash node 2 pieces
CPU=8
RAM=32 Gb
HDD= 300 Gb
The elasticsearch service is installed on 10 servers (2 kibana, 4 database, 2 ingest, 3 master)
- two shards are old templates, so previously there were only two data nodes.
How many shards do you recommend using for this cluster configuration?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 8, 2023, 12:10pm UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/11 "2023-11-08T12:10:38Z")
May I suggest you look at the following resource about sizing:
> **[Size your shards | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html)**
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [November 8, 2023, 12:26pm UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/12 "2023-11-08T12:26:39Z")
[dadoonet](https://discuss.elastic.co/u/dadoonet) ,
Thank you for the information, I’ll try to study it, but as always, on the official website everything is not always clear and unambiguously written 🙂 this is not just my opinion
Can you give me any advice on how to troubleshoot errors? so they've become very frequent lately
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [November 21, 2023, 1:37pm UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/13 "2023-11-21T13:37:15Z")
hi team. Any ideas why there might be such problems?
```auto
Nov 21 13:23:40 v-elk-lst02.my logstash[3389]: [2023-11-21T13:23:40,305][ERROR][logstash.outputs.elasticsearch][winlogbeat-pipe][winlogbeat-general] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://v-elk-eig01.my:9200/_bulk][Manticore::SocketTimeout] Read timed out", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
Nov 21 13:23:41 v-elk-lst02.my logstash[3389]: [2023-11-21T13:23:41,606][WARN][logstash.outputs.elasticsearch][wifi-dc1-pipe][1e38dc6fa9316ed39ff306c29b7fc695de9f516f3350520e99a10fbc1f5dbdf1] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://v-elk-eig01.my:9200/_bulk][Manticore::SocketTimeout] Read timed out {:url=>https://_internal:xxxxxx@v-elk-eig01.my:9200/, :error_message=>"Elasticsearch Unreachable: [https://v-elk-eig01.my:9200/_bulk][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
```
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [December 5, 2023, 9:05am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/14 "2023-12-05T09:05:20Z")
Hello team, I can’t fix this problem already and checked it over the network, but I didn’t find any blockages. I also noticed that when there is only one master left on the network, the cluster stops responding normally to requests. Where else should I look and what parameters should I check?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 5, 2023, 9:53am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/15 "2023-12-05T09:53:37Z")
The `Read timed out` indicates a timeout which is under Logstash's control. I suggest you ask in the Logstash forum about how to dig deeper or relax it.
> [@San9](#):
>
> I also noticed that when there is only one master left on the network, the cluster stops responding normally to requests.
That's the [expected behaviour](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-quorums.html):
> To be sure that the cluster remains available you **must not stop half or more of the nodes in the voting configuration at the same time**. As long as more than half of the voting nodes are available the cluster can still work normally. This means that if there are three or four master-eligible nodes, the cluster can tolerate one of them being unavailable.
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [December 5, 2023, 10:31am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/16 "2023-12-05T10:31:00Z")
I just see errors in the Elastic logs. of such a type:
```auto
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
```
At the same time, at this moment I cannot get the state of the cluster when connecting to one of the ingest nodes.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 5, 2023, 10:40am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/17 "2023-12-05T10:40:23Z")
There will be more info about that in the Elasticsearch logs.
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [December 5, 2023, 10:54am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/18 "2023-12-05T10:54:54Z")
This is what I see in the magazine, but I couldn’t understand what to do next...
I see that because of this problem, the logs do not always get into elastic in time.
v-elk-eig02 logs:
```auto
[2023-10-27T12:04:05,041][WARN][o.e.c.c.ClusterFormationFailureHelper] [v-elk-eig02.my] master not discovered yet: have discovered [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]; discovery will continue using [10.1.5.210:9301, 10.1.5.211:9301, 10.1.5.212:9301] from hosts providers and [{v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] from last-known cluster state; node term 45, last-accepted version 449311 in term 45; joining [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] in term [45] has status [waiting for response] after [14.9m/894401ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/discovery-troubleshooting.html
[2023-10-27T12:04:07,346][WARN][r.suppressed] [v-elk-eig02.my] path: /_monitoring/bulk, params: {system_id=logstash, system_api_version=7, interval=1s}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:174) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:60) ~[?:?]
at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:35) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:455) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1015) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:981) ~[?:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:469) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:185) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:459) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:435) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:322) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:150) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$4(CompositeRolesStore.java:194) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRole$5(CompositeRolesStore.java:212) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$0(RoleReferenceIntersection.java:49) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:56) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildRoleFromRoleReference(CompositeRolesStore.java:292) ~[?:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.lambda$buildRole$1(RoleReferenceIntersection.java:53) ~[?:?]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.elasticsearch.xpack.core.security.authz.store.RoleReferenceIntersection.buildRole(RoleReferenceIntersection.java:53) ~[?:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210) ~[?:?]
at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:187) ~[?:?]
at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:146) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:338) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:94) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:261) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:114) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:199) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:113) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:91) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:379) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$prepareRequest$0(RestMonitoringBulkAction.java:101) ~[?:?]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:96) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:87) ~[?:?]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.lambda$authenticateAndAttachToContext$3(SecondaryAuthenticator.java:99) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticate(SecondaryAuthenticator.java:109) ~[?:?]
at org.elasticsearch.xpack.security.authc.support.SecondaryAuthenticator.authenticateAndAttachToContext(SecondaryAuthenticator.java:90) ~[?:?]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:82) ~[?:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:415) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:544) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:317) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:453) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:549) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:426) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardData(Netty4HttpHeaderValidator.java:194) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.forwardFullRequest(Netty4HttpHeaderValidator.java:137) ~[?:?]
at org.elasticsearch.http.netty4.Netty4HttpHeaderValidator.lambda$requestStart$1(Netty4HttpHeaderValidator.java:120) ~[?:?]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[?:?]
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
[2023-10-27T12:04:08,602][WARN][o.e.x.m.MonitoringService] [v-elk-eig02.my] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$doFlush$0(ExportBulk.java:110) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$1(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:171) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:308) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:124) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.retry(TransportBulkAction.java:657) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:644) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:513) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
... 33 more
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:640) ~[elasticsearch-8.10.2.jar:?]
... 10 more
[2023-10-27T12:04:09,245][WARN][r.suppressed] [v-elk-eig02.my] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:640) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:513) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
[2023-10-27T12:04:15,041][WARN][o.e.c.c.ClusterFormationFailureHelper] [v-elk-eig02.my] master not discovered yet: have discovered [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]; discovery will continue using [10.1.5.210:9301, 10.1.5.211:9301, 10.1.5.212:9301] from hosts providers and [{v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] from last-known cluster state; node term 45, last-accepted version 449311 in term 45; joining [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] in term [45] has status [waiting for response] after [15m/904405ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/discovery-troubleshooting.html
[2023-10-27T12:04:16,763][WARN][r.suppressed] [v-elk-eig02.my] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:640) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:513) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.10.2.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.10.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
[2023-10-27T12:04:17,531][INFO][o.e.t.TcpTransport] [v-elk-eig02.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.6.14:49642, remoteAddress=10.1.5.212/10.1.5.212:9301, profile=default}], disconnecting from relevant node: Connection timed out
[2023-10-27T12:04:17,531][INFO][o.e.t.ClusterConnectionManager] [v-elk-eig02.my] transport connection to [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] closed by remote
[2023-10-27T12:04:17,531][INFO][o.e.c.c.JoinHelper] [v-elk-eig02.my] failed to join {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0} with JoinRequest{sourceNode={v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, transportVersion=8500061, minimumTerm=45, optionalJoin=Optional[Join{term=45, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}, targetNode={v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}}]}
org.elasticsearch.transport.NodeDisconnectedException: [v-elk-em03.my][10.1.5.212:9301][internal:cluster/coordination/join] disconnected
[2023-10-27T12:04:18,269][INFO][o.e.c.s.ClusterApplierService] [v-elk-eig02.my] master node changed {previous [], current [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]}, term: 45, version: 449312, reason: ApplyCommitRequest{term=45, version=449312, sourceNode={v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}{ml.config_version=10.0.0, xpack.installed=true, transform.config_version=10.0.0}}
```
Master logs v-elk-em03:
```auto
[2023-10-27T10:00:16,238][INFO][o.e.c.m.MetadataMappingService] [v-elk-em03.my] [my-ngfw-logs-2023.10.27/0En_oXU7Rrij4nY_REyT0g] update_mapping [_doc]
[2023-10-27T10:23:20,459][INFO][o.e.c.c.NodeJoinExecutor] [v-elk-em03.my] node-join: [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}] with reason [rejoining]
[2023-10-27T10:28:24,766][INFO][o.e.t.TcpTransport] [v-elk-em03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.212:9301, remoteAddress=/10.1.6.14:54406, profile=default}], disconnecting from relevant node: Connection reset
[2023-10-27T10:42:00,611][INFO][o.e.c.m.MetadataMappingService] [v-elk-em03.my] [my-ngfw-logs-2023.10.27/0En_oXU7Rrij4nY_REyT0g] update_mapping [_doc]
[2023-10-27T11:36:07,447][INFO][o.e.c.m.MetadataMappingService] [v-elk-em03.my] [winlogbeat-srv-sdc-2023.10.27/p1Hi2Sx6SamnWsJrFCBM4A] update_mapping [_doc]
[2023-10-27T11:36:54,642][INFO][o.e.c.c.NodeJoinExecutor] [v-elk-em03.my] node-join: [{v-elk-eig01.my}{pZK4-zUxTzGbfKIcLFVkwg}{BOjq36m9Tsm98rZo4Huxpw}{v-elk-eig01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.10.2}{7000099-8100299}] with reason [rejoining]
[2023-10-27T11:41:52,062][INFO][o.e.t.TcpTransport] [v-elk-em03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.212:9301, remoteAddress=/10.1.6.13:35226, profile=default}], disconnecting from relevant node: Connection reset
2023-10-27T12:04:18,283][INFO][o.e.c.c.NodeJoinExecutor] [v-elk-em03.my] node-join: [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}] with reason [rejoining]
[2023-10-27T12:09:26,847][INFO][o.e.t.TcpTransport] [v-elk-em03.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.212:9301, remoteAddress=/10.1.6.14:49642, profile=default}], disconnecting from relevant node: Connection reset
```
next step v-elk-em01:
```auto
[2023-10-27T02:20:31,153][INFO][o.e.t.TcpTransport] [v-elk-em01.my] close connection exception caught on transport layer [Netty4TcpChannel{localAddress=/10.1.5.210:9301, remoteAddress=/10.1.6.13:49494, profile=default}], disconnecting from relevant node: Connection reset
[2023-10-27T09:06:26,575][WARN][o.e.c.c.ClusterBootstrapService] [v-elk-em01.my] this node is locked into cluster UUID [ACjIGY_AT-6yr0tCdySHqw] but [cluster.initial_master_nodes] is set to [v-elk-em01.my, v-elk-em02.my, v-elk-em03.my]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/important-settings.html#initial_master_nodes
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 5, 2023, 10:55am UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/19 "2023-12-05T10:55:56Z")
> [@San9](#):
>
> `[2023-10-27T12:04:05,041][WARN][o.e.c.c.ClusterFormationFailureHelper] [v-elk-eig02.my] master not discovered yet: have discovered [{v-elk-eig02.my}{cV10VSK1STS60hWeyGqtcQ}{zu9h2c0_RZSeHkZxfT7z2w}{v-elk-eig02.my}{10.1.6.14}{10.1.6.14:9301}{i}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}]; discovery will continue using [10.1.5.210:9301, 10.1.5.211:9301, 10.1.5.212:9301] from hosts providers and [{v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{EumqtzHARGi7wmGKzdfT4Q}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em02.my}{8FaQnG_TT6uDoxRMB_8duw}{SwEk1aPXTa6B5YrlLSKnEg}{v-elk-em02.my}{10.1.5.211}{10.1.5.211:9301}{m}{8.10.2}{7000099-8100299}, {v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] from last-known cluster state; node term 45, last-accepted version 449311 in term 45; joining [{v-elk-em03.my}{xKXXQhmVT9efjGWQKQ-iGw}{k4A2is3IRiiuTGDfF9xQig}{v-elk-em03.my}{10.1.5.212}{10.1.5.212:9301}{m}{8.10.2}{7000099-8100299}] in term [45] has status [waiting for response] after [14.9m/894401ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/discovery-troubleshooting.html`
As it says:
> `for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/discovery-troubleshooting.html`
---
### Author:  [@San9](https://discuss.elastic.co/u/San9)
#### Post date: [December 19, 2023, 2:14pm UTC](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931/20 "2023-12-19T14:14:15Z")
I tried to follow this instruction, but the problem appears periodically, it is not constant, but quite often:
here is the new message that I saw in the logs:
```auto
[2023-12-19T10:37:16,950][INFO][o.e.c.c.JoinHelper] [v-elk-eig01.my] failed to join {v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{79ieu-v5QGOd8U0ByPhOGw}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true} with JoinRequest{sourceNode={v-elk-eig01.my}{pZK4-zUxTzGbfKIcLFVkwg}{_Xmrrx-aQTarFjR68BA9-w}{v-elk-eig01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true}, compatibilityVersions=CompatibilityVersions[transportVersion=8512001, systemIndexMappingsVersion={.triggered_watches=MappingsVersion[version=1, hash=-502826165], .secrets-inference=MappingsVersion[version=1, hash=-1434574148], .fleet-agents-7=MappingsVersion[version=1, hash=-644511608], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .ml-config=MappingsVersion[version=1, hash=1455005892], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=1, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-1=MappingsVersion[version=1, hash=1647955624], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-840564854], .fleet-actions-7=MappingsVersion[version=1, hash=961753072], .tasks=MappingsVersion[version=0, hash=-945584329], .ml-meta=MappingsVersion[version=1, hash=976416370], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .security-7=MappingsVersion[version=1, hash=-1061511639], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=2, hash=-1534099917], .inference=MappingsVersion[version=1, hash=645017346], .async-search=MappingsVersion[version=0, hash=-1403744380], .fleet-policies-7=MappingsVersion[version=1, hash=762674907], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], minimumTerm=53, optionalJoin=Optional[Join{term=53, lastAcceptedTerm=52, lastAcceptedVersion=524101, sourceNode={v-elk-eig01.my}{pZK4-zUxTzGbfKIcLFVkwg}{_Xmrrx-aQTarFjR68BA9-w}{v-elk-eig01.my}{10.1.6.13}{10.1.6.13:9301}{it}{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true}, targetNode={v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{79ieu-v5QGOd8U0ByPhOGw}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.11.1}{7000099-8500003}{ml.config_version=11.0.0, transform.config_version=10.0.0, xpack.installed=true}}]}
org.elasticsearch.transport.NodeDisconnectedException: [v-elk-em01.my][10.1.5.210:9301][internal:cluster/coordination/join] disconnected
```
and
```auto
[2023-12-19T14:24:49,556][WARN][r.suppressed] [v-elk-eig02.my] path: /_bulk, params: {}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
```
and next message
```auto
[2023-12-19T14:24:50,499][INFO][o.e.c.s.ClusterApplierService] [v-elk-eig02.my] master node changed {previous [], current [{v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{79ieu-v5QGOd8U0ByPhOGw}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.11.1}{7000099-8500003}]}, term: 53, version: 525188, reason: ApplyCommitRequest{term=53, version=525188, sourceNode={v-elk-em01.my}{J5e2urQZT0enBQgEtqxXSA}{79ieu-v5QGOd8U0ByPhOGw}{v-elk-em01.my}{10.1.5.210}{10.1.5.210:9301}{m}{8.11.1}{7000099-8500003}{xpack.installed=true, ml.config_version=11.0.0, transform.config_version=10.0.0}}
``
```
[Next page](https://discuss.elastic.co/t/unstable-operation-of-elasticsearch/345931.md?page=2)