# Elasticsearch 7.7 - all shards failed and task cancelled with channel closed **URL:** \ **Category:** Elasticsearch\ **Created:** [January 14, 2021, 10:11pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-all-shards-failed-and-task-cancelled-with-channel-closed/261184 "2021-01-14T22:11:12Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jamesp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamesp/32/73041_2.png) [@jamesp](https://discuss.elastic.co/u/jamesp)\ **Post date:** [January 14, 2021, 10:11pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-all-shards-failed-and-task-cancelled-with-channel-closed/261184/1 "2021-01-14T22:11:12Z")
Hello all, I set up a 3 node cluster using the included JRE on Windows 2016 servers with 12 GB RAM (java min/max 6GB) which seemed to be running fine without a lot of traffic. After hooking it up to a search GUI I started seeing this error: ```auto [2021-01-04T01:24:26,346][WARN][o.e.m.j.JvmGcMonitorService] [node1] [gc][381772] overhead, spent [7s] collecting in the last [7.3s] [2021-01-04T01:24:26,153][WARN][r.suppressed] [node1] path: /alias1,alias2/_search, params: {index=alias1,alias2} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:231) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:119) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.CountedCollector.countDown(CountedCollector.java:53) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.CountedCollector.onFailure(CountedCollector.java:76) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.FetchSearchPhase$2.onFailure(FetchSearchPhase.java:198) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:402) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1248) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1222) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:795) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService.access$100(TransportService.java:75) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:128) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:707) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:621) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:672) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:664) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.SearchTransportService.sendExecuteFetch(SearchTransportService.java:172) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.SearchTransportService.sendExecuteFetch(SearchTransportService.java:162) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.FetchSearchPhase.executeFetch(FetchSearchPhase.java:180) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:162) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:47) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:95) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.1.jar:7.7.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: org.elasticsearch.ElasticsearchException$1: Task cancelled before it started: channel closed at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:307) [elasticsearch-7.7.1.jar:7.7.1] ... 30 more Caused by: java.lang.IllegalStateException: Task cancelled before it started: channel closed at org.elasticsearch.tasks.TaskManager.registerCancellableTask(TaskManager.java:141) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.tasks.TaskManager.register(TaskManager.java:122) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:60) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:186) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:119) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676) ~[elasticsearch-7.7.1.jar:7.7.1] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:832) ~[?:?] ``` I've seen some other posts with this error but I have not seen a solution. Anyone have any ideas? Thanks! Jim ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 17, 2021, 10:54pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-all-shards-failed-and-task-cancelled-with-channel-closed/261184/2 "2021-01-17T22:54:47Z")
Welcome to our community! 😃 > [@jamesp](#): > > After hooking it up to a search GUI I started seeing this error: What GUI is this? Do those aliases/indices mentioned exist? ---
**Author:** ![nhat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nhat/32/22170_2.png) [@nhat](https://discuss.elastic.co/u/nhat)\ **Post date:** [January 18, 2021, 2:33am UTC](https://discuss.elastic.co/t/elasticsearch-7-7-all-shards-failed-and-task-cancelled-with-channel-closed/261184/3 "2021-01-18T02:33:42Z")
Hi @jamesp, Did you cancel an ongoing search request? Or the connection between your search GUI and Elasticsearch got disconnected? ---
**Author:** ![jamesp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamesp/32/73041_2.png) [@jamesp](https://discuss.elastic.co/u/jamesp)\ **Post date:** [January 18, 2021, 4:47pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-all-shards-failed-and-task-cancelled-with-channel-closed/261184/4 "2021-01-18T16:47:20Z")
Mark, Thanks for the reply. We are using a custom built javascript GUI. Those aliases are set up in the cluster and map back to indices with documents. I note that I have seen this error multiple times and have increased the RAM on the servers from 4 to 6 GB but still get the error, also it seems the system recovers and continues to return results. The cluster only contains about 1-2 millions documents at the moment. Jim ---
**Author:** ![jamesp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamesp/32/73041_2.png) [@jamesp](https://discuss.elastic.co/u/jamesp)\ **Post date:** [January 19, 2021, 7:43pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-all-shards-failed-and-task-cancelled-with-channel-closed/261184/5 "2021-01-19T19:43:37Z")
Nhat, Thanks for the reply, I seen a drop in connection and I've tried to replicate one but have not succeeded. Jim ---
**Author:** ![jamesp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamesp/32/73041_2.png) [@jamesp](https://discuss.elastic.co/u/jamesp)\ **Post date:** [January 19, 2021, 7:57pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-all-shards-failed-and-task-cancelled-with-channel-closed/261184/6 "2021-01-19T19:57:24Z")
I will add that I moved from version 6.2.3 to 7.7.1. The two indexes linked to the aliases listed were indexed via _reindex from a remote cluster_, I created a new empty index, then indexed the remote data into each newly created index in the 7.7 cluster. We are also using readonlyrest, I had some issues getting the basic X-Pack package to work. Jim ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 16, 2021, 7:57pm UTC](https://discuss.elastic.co/t/elasticsearch-7-7-all-shards-failed-and-task-cancelled-with-channel-closed/261184/7 "2021-02-16T19:57:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.