Search Requests - How threads work

[chindhu_hari]

@jpountz @jprante

I hit the following exception in my ES. It's a 2.3.2 with single node, 32 CPU linux machine. 5 shards per index. I'm querying on a particular index.
I have few queries based upon the log entry below:

1. queued tasks = 1000 <-- Does this number refer to only the incoming search requests or the total number of internal tasks that might have got initiated during a single search query (like fetch shard et al).

2. active threads = 49 <-- Does this mean, for each search request a thread is assigned or is it for each shard? Say, my index has 5 shards and a search request is made against this index, does this mean 5 threads will allocated for this request or just 1?

   Failed to execute fetch phase
   RemoteTransportException[[plefg18][plefg18.us.oracle.com/10.147.34.35:9300][indices:data/read/search[phase/fetch/id]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$4@5d74cdb2 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@55c9ea6a[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 13953423]]];

Also, below, how does the fetch_total and query_total get calculated? For every search request of mine, I was expecting them to increase by 5(since it'll be hitting 5 shards), query_total was increasing by 5, but fetch_total gets increased by 1.

{search: {open_contexts: 0,**query_total: 314575**,query_time_in_millis: 6562305,query_current: 0,**fetch_total: 42058**,fetch_time_in_millis: 161747,fetch_current: 0,scroll_total: 0,scroll_time_in_millis: 0,scroll_current: 0}

[dakrone (Lee Hinman)]

@chindhu_hari

1. queued tasks = 1000 <-- Does this number refer to only the incoming search requests or the total number of internal tasks that might have got initiated during a single search query (like fetch shard et al).

So this is any task that is on the "search" threadpool. For example, in a single node case querying an index with 5 primary shards, you will end up with 6 total tasks (1 coordinating task and 1 task for executing the query on each shard).

@chindhu_hari

2. active threads = 49 <-- Does this mean, for each search request a thread is assigned or is it for each shard? Say, my index has 5 shards and a search request is made against this index, does this mean 5 threads will allocated for this request or just 1?

The active_threads are the number of threads currently processing a task in this particular threadpool. It is a per-node threadpool. If you have 5 shards on the node and a query is sent against all of them, that means 5 threads can work on executing the query (see the explanation above)

Hope this helps