Elasticsearch Error "EsRejectedExecutionException"

Elastic Stack Elasticsearch
1

Hi Experts,

My ES1.7.1 logs are full with the following error , and due to this I am not able to open Kibana Page it is hang forever.

[2016-04-21 03:51:55,334][DEBUG][action.search.type       ] [Tarantula] [cef-2016-04-03][2], node[t-mO7uX-Tla2DCjLcJgF0w], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3c1bf284] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Ned Horrocks][inet[/172.24.11.70:10001]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@694201c0
        at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
        at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:79)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:224)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:114)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Please help me to understand how I can resolve this issue

2

When you see this sort of message

rejected execution (queue capacity 1000)

it is an indication that all threads are busy servicing requests, there is also a queue of pending requests waiting for a free thread which is full meaning the latest request cannot be added to the waitlist because it will exceed the threshold of 1000.

Short answer - you've exceeded the configured compute capacity of that server and need to back off with your requests.

3

Thanks Mark,

But how i can back Off with my requests , i stopped LS, restarted ES , increased no of nodes also Increased threadpool.search.queue_size: 2000.

Even after doing above My kibana screen is hanged on the very first page where is says > "KIBANA is loading. Give me a moment here. I'm loading a whole bunch of code. Don't worry, all this good stuff will be cached up for next time!"

4

Think of queue sizes as ways of dealing with temporary spikes in request volumes. If your flow of traffic is constant and consistently exceeding the capacity of the thread pool expect queues to back up and overflow.

At this stage I'd get to the bottom of what the expensive queries are on your system if you've already been through a round of just adding more nodes. Check the slow query logs, consider trying the new Query Profiler to diagnose individual queries for the main culprits.

5

yes i can enable slow query log at the ES level , but the problem is Kibana is not opening any dashboard ,it is stuck on the first page . So how would i identify which dashboard query is slow . I am stuck !!!

6

So how would i identify which dashboard query is slow .

Browser dev tools can show a panel with pending network requests etc if you're trying to figure out the browser state.
Kibana log files will help with the Kibana server but you're best off asking these questions in the Kibana forum.
Elasticsearch has slow query logs, hot_threads and various CAT apis to show the state of the elastic servers. Marvel/monitoring will show a lot of these stats and OS level ones too like CPU and IO.

So there's a number of places to look. Just how many elasticsearch users are hitting your cluster?

7

Never Mind , I resolved the problem , issue was at the Kibana end so i take help from browser dev tool, Port was not responding which causes the issue . Well thanks again Mark, thing seems up now.

1 Like