# My ES has some problems "Failed to execute fetch phase "

**URL:** <https://discuss.elastic.co/t/my-es-has-some-problems-failed-to-execute-fetch-phase/118960>\
**Category:** Elasticsearch\
**Created:** [February 8, 2018, 3:36am UTC](https://discuss.elastic.co/t/my-es-has-some-problems-failed-to-execute-fetch-phase/118960 "2018-02-08T03:36:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hadogen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadogen/32/91070_2.png) [@hadogen](https://discuss.elastic.co/u/hadogen)\
**Post date:** [February 8, 2018, 3:36am UTC](https://discuss.elastic.co/t/my-es-has-some-problems-failed-to-execute-fetch-phase/118960/1 "2018-02-08T03:36:49Z")

</div>

heIp me !  
I have temporarily used two nodes to deploy elasticsearch-5.2.2 ， Now I have such a problem, I how to deal with this issue？

```
[2018-02-08T09:23:09,357][DEBUG][o.e.a.s.TransportSearchAction] [node-110] [195737742] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [node-111][192.168.20.111:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TcpTransport$RequestHandler@762a3357 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsea
rch.common.util.concurrent.EsThreadPoolExecutor@5fa58f86[Running, pool size = 13, active threads = 13, queued tasks = 1000, completed tasks = 305718666]]
	at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:50) ~[elasticsearch-5.2.2.jar:5.2.2]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_65]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) ~[?:1.8.0_65]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1445) [elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1329) [elasticsearch-5.2.2.jar:5.2.2]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-5.2.2.jar:5.2.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

```

help me ~ thanks very much !

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [February 8, 2018, 3:51am UTC](https://discuss.elastic.co/t/my-es-has-some-problems-failed-to-execute-fetch-phase/118960/2 "2018-02-08T03:51:41Z")

</div>

It seems you have too many search requests queued up and your nodes can't keep up with the pace you're setting.

You can read the exception message like this: Your search thread pool has 13 threads (`pool size = 13`) and all are busy (`active threads = 13`). By default, when all threads are active, all search requests coming in the meantime are queued waiting for a thread to be available to handle them. In your case, that queue is full (`queued tasks = 1000`). This exception is caused by the 1014th search request coming in (1000 waiting + 13 active being processed + this one). Increasing the queue size might work, but it's only postponing the problem.

This means that your 8 CPUs/cores nodes (the pool size is determined by the following equation: `int((# of available_processors * 3) / 2) + 1`) are not able to keep up with the load your client app is throwing at it. You might need to scale out (i.e. add nodes) in order to handle a higher capacity.

You'll need to provide more info concerning your indices, shards, the size of your data, heap size, etc.

---

<div class="post-metadata">

**Author:** ![hadogen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadogen/32/91070_2.png) [@hadogen](https://discuss.elastic.co/u/hadogen)\
**Post date:** [February 8, 2018, 9:33am UTC](https://discuss.elastic.co/t/my-es-has-some-problems-failed-to-execute-fetch-phase/118960/3 "2018-02-08T09:33:03Z")

</div>

Thank you ，You're awesome！

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 8, 2018, 9:33am UTC](https://discuss.elastic.co/t/my-es-has-some-problems-failed-to-execute-fetch-phase/118960/4 "2018-03-08T09:33:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
