# Kibana query Error loading gateway out

**URL:** <https://discuss.elastic.co/t/kibana-query-error-loading-gateway-out/313724>\
**Category:** Elasticsearch\
**Created:** [September 6, 2022, 2:11am UTC](https://discuss.elastic.co/t/kibana-query-error-loading-gateway-out/313724 "2022-09-06T02:11:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![liucx1201](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liucx1201/32/110496_2.png) [@liucx1201](https://discuss.elastic.co/u/liucx1201)\
**Post date:** [September 6, 2022, 2:11am UTC](https://discuss.elastic.co/t/kibana-query-error-loading-gateway-out/313724/1 "2022-09-06T02:11:22Z")

</div>

Recently, cold and hot data wind separation was operated。

Data writing is normal, but query error occurs in the afternoon of the third day

An error is reported through kibana query:Kibana query Error loading gateway out

View the log of the master node and report the following error:

```auto
2022-09-06T01:03:01,702][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [es-node-3] Sending remove ban for tasks with the parent [AiNzpm1mTHSf3neMD_Bp8A:213793856] to the node [76NzEVMgRQqVUIEgL2004A]
[2022-09-06T01:03:01,702][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [es-node-3] Sending remove ban for tasks with the parent [AiNzpm1mTHSf3neMD_Bp8A:213793856] to the node [sH4ugLUGQTeADT8oggUz5A]
[2022-09-06T01:03:01,702][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [es-node-3] Sending remove ban for tasks with the parent [AiNzpm1mTHSf3neMD_Bp8A:213793856] to the node [P-nmL3VhRTiFnFcuYO-jGg]
[2022-09-06T01:03:01,702][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [es-node-3] Sending remove ban for tasks with the parent [AiNzpm1mTHSf3neMD_Bp8A:213793856] to the node [JnGPvEBBT-Cg9DPcjg4qOA]
[2022-09-06T01:03:01,702][WARN][r.suppressed] [es-node-3] path: /.kibana/_search, params: {index=.kibana}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:314) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:273) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:424) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:243) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:241) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:233) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:136) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:663) [elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62) [transport-netty4-client-7.4.2.jar:7.4.2]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) [netty-codec-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) [netty-codec-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-common-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.38.Final.jar:4.1.38.Final]
        at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
        at org.elasticsearch.search.query.CancellableCollector.getLeafCollector(CancellableCollector.java:49) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.search.internal.ContextIndexSearcher.searchInternal(ContextIndexSearcher.java:181) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:174) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:440) ~[lucene-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe - ivera - 2019-07-19 15:05:56]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:270) ~[elasticsearch-7.4.2.jar:7.4.2]
                                                                                                                                                                 

```

The data cannot be queried in the hot node, and the writing is normal. After that, replace the data with a cold label to resume the query

Please help me have a look

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 6, 2022, 2:12am UTC](https://discuss.elastic.co/t/kibana-query-error-loading-gateway-out/313724/2 "2022-09-06T02:12:10Z")

</div>

Welcome to our community! 😃

What is the output from the [`_cluster/stats?pretty&human` API](https://www.elastic.co/guide/en/elasticsearch/reference/8.4/cluster-stats.html)?

---

<div class="post-metadata">

**Author:** ![liucx1201](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liucx1201/32/110496_2.png) [@liucx1201](https://discuss.elastic.co/u/liucx1201)\
**Post date:** [September 6, 2022, 2:40am UTC](https://discuss.elastic.co/t/kibana-query-error-loading-gateway-out/313724/4 "2022-09-06T02:40:43Z")

</div>

Warm node: es-node-1 --- node5 (5)  
Hot nodes: es-node-6 --- node8 (3)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 6, 2022, 3:29am UTC](https://discuss.elastic.co/t/kibana-query-error-loading-gateway-out/313724/5 "2022-09-06T03:29:22Z")

</div>

> [@liucx1201](#):
>
> ```auto
> "versions" : [
> "7.4.2"
> ],
> 
> ```

Please upgrade, that version is no longer supported and is [EOL](https://www.elastic.co/support/eol).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 4, 2022, 3:30am UTC](https://discuss.elastic.co/t/kibana-query-error-loading-gateway-out/313724/6 "2022-10-04T03:30:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.