Elasticsearch 7.17.10 stuck with "triggering scheduled [ML] maintenance tasks"

Elastic Stack Elasticsearch
1

Hi,

After using Elasticsearch on my laptop for quite a while without problems, I recently upgraded from 7.17.7 to 7.17.10. Since that, I encountered twice a total freeze/stuck behavior. First time I stopped and restarted the docker. Now it happened again.

The log show:

2023-06-08 16:15:56 {"type": "server", "timestamp": "2023-06-08T13:15:56,640Z", "level": "INFO", "component": "o.e.x.m.MlDailyMaintenanceService", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "triggering scheduled [ML] maintenance tasks", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ" }

What is the problem? and how to solve it.

Thanks.

2

i can login into the Elasticsearch terminal in docker. But http://localhost:9200 in the browser spins infinitely.

3

What do all the Elasticsearch logs show?

4

Not sure how to upload all the logs. Here are the last lines:

2023-06-07 16:17:25 {"type": "server", "timestamp": "2023-06-07T13:17:25,690Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "path: /.kibana_task_manager_7.17.10_001/_pit, params: {index=.kibana_task_manager_7.17.10_001, keep_alive=10m}", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ" , 
2023-06-07 16:17:25 "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:713) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:400) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:745) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:64) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:331) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService$4.handleException(TransportService.java:853) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1481) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1607) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1581) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:41) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.search.SearchService.lambda$openReaderContext$10(SearchService.java:965) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.index.shard.IndexShard.awaitShardSearchActive(IndexShard.java:4005) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.search.SearchService.openReaderContext(SearchService.java:945) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportOpenPointInTimeAction$ShardOpenReaderRequestHandler.messageReceived(TransportOpenPointInTimeAction.java:177) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportOpenPointInTimeAction$ShardOpenReaderRequestHandler.messageReceived(TransportOpenPointInTimeAction.java:174) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:1037) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService.access$100(TransportService.java:66) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:143) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:975) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:874) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:937) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:925) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportOpenPointInTimeAction.lambda$doExecute$0(TransportOpenPointInTimeAction.java:102) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportSearchAction$1$1.executePhaseOnShard(TransportSearchAction.java:347) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:316) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:352) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:244) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:199) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1048) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:305) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:91) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.TransportOpenPointInTimeAction.doExecute(TransportOpenPointInTimeAction.java:39) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:186) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:161) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.RestOpenPointInTimeAction.lambda$prepareRequest$0(RestOpenPointInTimeAction.java:45) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:109) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:337) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:403) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:255) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:35) [transport-netty4-client-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:19) [transport-netty4-client-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48) [transport-netty4-client-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]",
2023-06-07 16:17:25 "at java.lang.Thread.run(Thread.java:1623) [?:?]",
2023-06-07 16:17:25 "Caused by: org.elasticsearch.action.NoShardAvailableActionException: [28b95299d611][172.17.0.2:9300][indices:data/read/open_reader_context]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:532) ~[elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:479) [elasticsearch-7.17.10.jar:7.17.10]",
2023-06-07 16:17:25 "... 109 more"] }
2023-06-07 16:17:31 {"type": "server", "timestamp": "2023-06-07T13:17:31,789Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana_task_manager_7.17.10_001][0]]]).", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-07 16:18:30 {"type": "server", "timestamp": "2023-06-07T13:18:30,357Z", "level": "INFO", "component": "o.e.t.LoggingTaskListener", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "1052 finished with response BulkByScrollResponse[took=642.1ms,timed_out=false,sliceId=null,updated=17,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-07 16:18:33 {"type": "server", "timestamp": "2023-06-07T13:18:33,856Z", "level": "INFO", "component": "o.e.x.i.a.TransportPutLifecycleAction", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "updating index lifecycle policy [.alerts-ilm-policy]", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-07 16:19:34 {"type": "deprecation.elasticsearch", "timestamp": "2023-06-07T13:19:34,778Z", "level": "WARN", "component": "o.e.d.c.m.IndexNameExpressionResolver", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .kibana_7.17.10_001, .kibana_7.17.7_001, .kibana_task_manager_7.17.10_001, .kibana_task_manager_7.17.7_001, .tasks], but in a future major version, direct access to system indices will be prevented by default", "key": "open_system_index_access", "category": "api", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-07 16:20:28 {"type": "deprecation.elasticsearch", "timestamp": "2023-06-07T13:20:28,183Z", "level": "WARN", "component": "o.e.d.c.m.IndexNameExpressionResolver", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "this request accesses system indices: [.tasks], but in a future major version, direct access to system indices will be prevented by default", "key": "open_system_index_access", "category": "api", "elasticsearch.elastic_product_origin": "kibana", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-07 16:22:38 {"type": "server", "timestamp": "2023-06-07T13:22:38,594Z", "level": "INFO", "component": "o.e.m.j.JvmGcMonitorService", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "[gc][380] overhead, spent [291ms] collecting in the last [1s]", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-07 16:22:45 {"type": "server", "timestamp": "2023-06-07T13:22:45,185Z", "level": "INFO", "component": "o.e.m.j.JvmGcMonitorService", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "[gc][386] overhead, spent [661ms] collecting in the last [1.5s]", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-07 17:02:09 {"type": "server", "timestamp": "2023-06-07T14:02:09,122Z", "level": "WARN", "component": "o.e.m.j.JvmGcMonitorService", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "[gc][2726] overhead, spent [2.4s] collecting in the last [3s]", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-08 08:39:08 {"type": "server", "timestamp": "2023-06-08T05:39:08,200Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "timer thread slept for [13s/13008ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-08 08:39:08 {"type": "server", "timestamp": "2023-06-08T05:39:08,650Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "timer thread slept for [13s/13007716867ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-08 08:39:08 {"type": "server", "timestamp": "2023-06-08T05:39:08,855Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "timer thread slept for [12.6h/45561335ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-08 08:39:08 {"type": "server", "timestamp": "2023-06-08T05:39:08,856Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "timer thread slept for [5.4s/5414812143ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
2023-06-08 16:15:56 {"type": "server", "timestamp": "2023-06-08T13:15:56,640Z", "level": "INFO", "component": "o.e.x.m.MlDailyMaintenanceService", "cluster.name": "my-elasticsearch-cluster", "node.name": "28b95299d611", "message": "triggering scheduled [ML] maintenance tasks", "cluster.uuid": "uoed-H1gQHCO--NmtP3ckA", "node.id": "kWbQYB1tRzCF7tUPXLhAcQ"  }
5

From within the terminal I tried jstack:

sh-5.0# ps aux | grep java
elastic+     8 39.8 36.3 3832412 1083184 ?     Sl   Jun08 509:31 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-11639016333509815199 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Des.cgroups.hierarchy.override=/ -Xms1g -Xmx1g -XX:MaxDirectMemorySize=536870912 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Ediscovery.type=single-node
root      4600  0.0  0.0   3312   644 pts/0    S+   10:35   0:00 grep java
sh-5.0# jstack 8 > thread_dump.txt
8: Unable to open socket file /proc/8/root/tmp/.java_pid8: target process 8 doesn't respond within 10500ms or HotSpot VM not loaded
6

What is the output from the _cluster/stats?pretty&human API?
And the output from GET _cat/health?v

7

Unfortunately when this occurs I cannot execute APIs as http://localhost:9200 completely stuck.
Yesterday I restarted the container, and also removed replicas shards and a couple of redundant indices.

Now I executed the APIs:

epoch      timestamp cluster                  status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1686644230 08:17:10  my-elasticsearch-cluster green           1         1     50  50    0    0        0             0                  -                100.0%


{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "my-elasticsearch-cluster",
  "cluster_uuid" : "uoed-H1gQHCO--NmtP3ckA",
  "timestamp" : 1686644156900,
  "status" : "green",
  "indices" : {
    "count" : 30,
    "shards" : {
      "total" : 50,
      "primaries" : 50,
      "replication" : 0.0,
      "index" : {
        "shards" : {
          "min" : 1,
          "max" : 5,
          "avg" : 1.6666666666666667
        },
        "primaries" : {
          "min" : 1,
          "max" : 5,
          "avg" : 1.6666666666666667
        },
        "replication" : {
          "min" : 0.0,
          "max" : 0.0,
          "avg" : 0.0
        }
      }
    },
    "docs" : {
      "count" : 4555,
      "deleted" : 22819
    },
    "store" : {
      "size_in_bytes" : 93426943,
      "total_data_set_size_in_bytes" : 93426943,
      "reserved_in_bytes" : 0
    },
    "fielddata" : {
      "memory_size_in_bytes" : 0,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size_in_bytes" : 4056,
      "total_count" : 190791,
      "hit_count" : 6707,
      "miss_count" : 184084,
      "cache_size" : 2,
      "cache_count" : 155,
      "evictions" : 153
    },
    "completion" : {
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 101,
      "memory_in_bytes" : 594532,
      "terms_memory_in_bytes" : 502176,
      "stored_fields_memory_in_bytes" : 49800,
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory_in_bytes" : 15680,
      "points_memory_in_bytes" : 0,
      "doc_values_memory_in_bytes" : 26876,
      "index_writer_memory_in_bytes" : 295004,
      "version_map_memory_in_bytes" : 358,
      "fixed_bit_set_memory_in_bytes" : 8064,
      "max_unsafe_auto_id_timestamp" : 1685958716561,
      "file_sizes" : { }
    },
    "mappings" : {
      "field_types" : [
        {
          "name" : "boolean",
          "count" : 127,
          "index_count" : 16,
          "script_count" : 0
        },
        {
          "name" : "constant_keyword",
          "count" : 6,
          "index_count" : 2,
          "script_count" : 0
        },
        {
          "name" : "date",
          "count" : 124,
          "index_count" : 22,
          "script_count" : 0
        },
        {
          "name" : "float",
          "count" : 8,
          "index_count" : 4,
          "script_count" : 0
        },
        {
          "name" : "integer",
          "count" : 100,
          "index_count" : 13,
          "script_count" : 0
        },
        {
          "name" : "ip",
          "count" : 2,
          "index_count" : 2,
          "script_count" : 0
        },
        {
          "name" : "keyword",
          "count" : 974,
          "index_count" : 22,
          "script_count" : 0
        },
        {
          "name" : "long",
          "count" : 100,
          "index_count" : 20,
          "script_count" : 0
        },
        {
          "name" : "nested",
          "count" : 66,
          "index_count" : 17,
          "script_count" : 0
        },
        {
          "name" : "object",
          "count" : 82,
          "index_count" : 12,
          "script_count" : 0
        },
        {
          "name" : "search_as_you_type",
          "count" : 36,
          "index_count" : 9,
          "script_count" : 0
        },
        {
          "name" : "text",
          "count" : 40,
          "index_count" : 22,
          "script_count" : 0
        },
        {
          "name" : "version",
          "count" : 4,
          "index_count" : 4,
          "script_count" : 0
        }
      ],
      "runtime_field_types" : [ ]
    },
    "analysis" : {
      "char_filter_types" : [ ],
      "tokenizer_types" : [ ],
      "filter_types" : [ ],
      "analyzer_types" : [ ],
      "built_in_char_filters" : [ ],
      "built_in_tokenizers" : [ ],
      "built_in_filters" : [ ],
      "built_in_analyzers" : [ ]
    },
    "versions" : [
      {
        "version" : "7.17.7",
        "index_count" : 15,
        "primary_shard_count" : 19,
        "total_primary_bytes" : 85251073
      },
      {
        "version" : "7.17.10",
        "index_count" : 15,
        "primary_shard_count" : 31,
        "total_primary_bytes" : 8175870
      }
    ]
  },
  "nodes" : {
    "count" : {
      "total" : 1,
      "coordinating_only" : 0,
      "data" : 1,
      "data_cold" : 1,
      "data_content" : 1,
      "data_frozen" : 1,
      "data_hot" : 1,
      "data_warm" : 1,
      "ingest" : 1,
      "master" : 1,
      "ml" : 1,
      "remote_cluster_client" : 1,
      "transform" : 1,
      "voting_only" : 0
    },
    "versions" : [
      "7.17.10"
    ],
    "os" : {
      "available_processors" : 2,
      "allocated_processors" : 2,
      "names" : [
        {
          "name" : "Linux",
          "count" : 1
        }
      ],
      "pretty_names" : [
        {
          "pretty_name" : "Ubuntu 20.04.6 LTS",
          "count" : 1
        }
      ],
      "architectures" : [
        {
          "arch" : "amd64",
          "count" : 1
        }
      ],
      "mem" : {
        "total_in_bytes" : 3054694400,
        "free_in_bytes" : 108089344,
        "used_in_bytes" : 2946605056,
        "free_percent" : 4,
        "used_percent" : 96
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 16
      },
      "open_file_descriptors" : {
        "min" : 540,
        "max" : 540,
        "avg" : 540
      }
    },
    "jvm" : {
      "max_uptime_in_millis" : 27020661,
      "versions" : [
        {
          "version" : "20.0.1",
          "vm_name" : "OpenJDK 64-Bit Server VM",
          "vm_version" : "20.0.1+9-29",
          "vm_vendor" : "Oracle Corporation",
          "bundled_jdk" : true,
          "using_bundled_jdk" : true,
          "count" : 1
        }
      ],
      "mem" : {
        "heap_used_in_bytes" : 388883848,
        "heap_max_in_bytes" : 1073741824
      },
      "threads" : 67
    },
    "fs" : {
      "total_in_bytes" : 486856978432,
      "free_in_bytes" : 283536658432,
      "available_in_bytes" : 283536658432
    },
    "plugins" : [ ],
    "network_types" : {
      "transport_types" : {
        "netty4" : 1
      },
      "http_types" : {
        "netty4" : 1
      }
    },
    "discovery_types" : {
      "single-node" : 1
    },
    "packaging_types" : [
      {
        "flavor" : "default",
        "type" : "docker",
        "count" : 1
      }
    ],
    "ingest" : {
      "number_of_pipelines" : 1,
      "processor_stats" : {
        "gsub" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time_in_millis" : 0
        },
        "script" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time_in_millis" : 0
        }
      }
    }
  }
}

At this point I am awaiting for it to freeze again. I think this will happen after 24 hours when the ML maintenance message will show again. This time I hope to be able to get jstacks to work.

8

You seem to have about 1 gig in heap, which is very low and I would start by increasing that to at least 2 gig.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.