Best practice for thread pool queue size

Zaki_Agha · October 13, 2014, 9:16pm

Hi
We have several elastic search clusters
Recently we faced an issue in which one of our nodes experienced queueing.
In fact, the queue length was greater than 1000.
Subsequent requests were rejected as the queue was full.

Should we increase the default queue size?

I understand that there are several queue's within elastic search.

Queues in Elastic Search
1.

  Index 
  <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html#modules-threadpool>           - 
  default 200
  2. 
  
  Bulk              - default 50
  3. 
  
  Get               - default 1000
  4. 
  
  Search         - default 1000
  5. 
  
  Suggest       - default 1000
  6. 
  
  Percolate     - default 1000
  7. 
  
  ThreadPool queue_size 
  <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html#_literal_fixed_literal>: 
   1000

Errors:

Error # 1

[[LApp45][SiyuJOHVRRG1udLiFwM9Yw][es1][inet[/xxx.xxx.xxx.xxx:9300]]], id
[84124759]

[2014-07-13 04:13:35,332][WARN ][transport ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[2014-07-13 04:13:35,332][WARN ][transport ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[[LApp37][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id
[80874233]

Error # 2

[2014-07-13 06:28:26,043][WARN ]

[transport ]

[es2] Received response for a request that has timed out,

sent [55795ms] ago,

timed out [25795ms] ago,

action [discovery/zen/fd/ping],

node

Error # 3

[2014-07-13 06:28:26,049][WARN ][transport ]

[es2] Received response for a request that has timed out,

sent [56023ms] ago,

timed out [26023ms] ago,

action [discovery/zen/fd/ping],

node [[es3][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id
[84124758]

Error # 4

There are several errors of this type all for the same index
aggregated_user_game_points

[2014-07-13 06:28:26,153][DEBUG][action.search.type ]

[es2] [aggregated_user_game_points][3], node[8qI5LGo2TxG1S-mQUgEA_w], [P], s

[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3367563e]
lastShard [true]

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected
execution (queue capacity 1000) on

org.elasticsearch.action.search
.type.TransportSearchTypeAction$BaseAsyncAction$4@71bd1bf

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · October 13, 2014, 10:21pm

Increasing queues isn't going to help if there are underlying problems
stopping the processing.

Based on those errors it looks like you may have network issues, but they
are from July?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 14 October 2014 08:16, Zaki Agha zaki@roblox.com wrote:

Hi
We have several Elasticsearch clusters
Recently we faced an issue in which one of our nodes experienced queueing.
In fact, the queue length was greater than 1000.
Subsequent requests were rejected as the queue was full.

Should we increase the default queue size?

I understand that there are several queue's within Elasticsearch.

Queues in Elastic Search
1.
  Index
  <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html#modules-threadpool>           -
  default 200
  2.

  Bulk              - default 50
  3.

  Get               - default 1000
  4.

  Search         - default 1000
  5.

  Suggest       - default 1000
  6.

  Percolate     - default 1000
  7.

  ThreadPool queue_size
  <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html#_literal_fixed_literal>:
   1000
Errors:

Error # 1

[[LApp45][SiyuJOHVRRG1udLiFwM9Yw][es1][inet[/xxx.xxx.xxx.xxx:9300]]], id
[84124759]

[2014-07-13 04:13:35,332][WARN ][transport ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[2014-07-13 04:13:35,332][WARN ][transport ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[[LApp37][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id
[80874233]

Error # 2

[2014-07-13 06:28:26,043][WARN ]

[transport ]

[es2] Received response for a request that has timed out,

sent [55795ms] ago,

timed out [25795ms] ago,

action [discovery/zen/fd/ping],

node

Error # 3

[2014-07-13 06:28:26,049][WARN ][transport ]

[es2] Received response for a request that has timed out,

sent [56023ms] ago,

timed out [26023ms] ago,

action [discovery/zen/fd/ping],

node [[es3][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]],
id [84124758]

Error # 4

There are several errors of this type all for the same index
aggregated_user_game_points

[2014-07-13 06:28:26,153][DEBUG][action.search.type ]

[es2] [aggregated_user_game_points][3], node[8qI5LGo2TxG1S-mQUgEA_w],
[P], s

[STARTED]: Failed to execute [org.elasticsearch.action.search
.SearchRequest@3367563e] lastShard [true]

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected
execution (queue capacity 1000) on

org.elasticsearch.action.search
.type.TransportSearchTypeAction$BaseAsyncAction$4@71bd1bf

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YdD2vWb-aPj2ySqR%2B%3D8K0fH7NTFjHjF426Gvcmj4mKUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Zaki_Agha · October 17, 2014, 11:51pm

Yes the particular error is from July.
How can I determine the optimal setting for queue size?

On Monday, October 13, 2014 3:21:32 PM UTC-7, Mark Walkom wrote:

Increasing queues isn't going to help if there are underlying problems
stopping the processing.

Based on those errors it looks like you may have network issues, but they
are from July?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 14 October 2014 08:16, Zaki Agha <za...@roblox.com <javascript:>>
wrote:
Hi
We have several Elasticsearch clusters
Recently we faced an issue in which one of our nodes experienced queueing.
In fact, the queue length was greater than 1000.
Subsequent requests were rejected as the queue was full.

Should we increase the default queue size?

I understand that there are several queue's within Elasticsearch.

Queues in Elastic Search
1.
  Index 
  <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html#modules-threadpool>           - 
  default 200
  2. 
  
  Bulk              - default 50
  3. 
  
  Get               - default 1000
  4. 
  
  Search         - default 1000
  5. 
  
  Suggest       - default 1000
  6. 
  
  Percolate     - default 1000
  7. 
  
  ThreadPool queue_size 
  <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html#_literal_fixed_literal>: 
   1000
Errors:

Error # 1

[[LApp45][SiyuJOHVRRG1udLiFwM9Yw][es1][inet[/xxx.xxx.xxx.xxx:9300]]], id
[84124759]

[2014-07-13 04:13:35,332][WARN ][transport ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[2014-07-13 04:13:35,332][WARN ][transport ]

[es2] Received response for a request that has timed out,

sent [55372ms] ago,

timed out [25372ms] ago,

action [discovery/zen/fd/ping],

node

[[LApp37][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]], id
[80874233]

Error # 2

[2014-07-13 06:28:26,043][WARN ]

[transport ]

[es2] Received response for a request that has timed out,

sent [55795ms] ago,

timed out [25795ms] ago,

action [discovery/zen/fd/ping],

node

Error # 3

[2014-07-13 06:28:26,049][WARN ][transport ]

[es2] Received response for a request that has timed out,

sent [56023ms] ago,

timed out [26023ms] ago,

action [discovery/zen/fd/ping],

node [[es3][FKVv20F4RSiEsxJ4Bo8rMA][es3][inet[/xxx.xxx.xxx.xxx:9300]]],
id [84124758]

Error # 4

There are several errors of this type all for the same index
aggregated_user_game_points

[2014-07-13 06:28:26,153][DEBUG][action.search.type ]

[es2] [aggregated_user_game_points][3], node[8qI5LGo2TxG1S-mQUgEA_w],
[P], s

[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3367563e]
lastShard [true]

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected
execution (queue capacity 1000) on

org.elasticsearch.action.search
.type.TransportSearchTypeAction$BaseAsyncAction$4@71bd1bf

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9cc3b7a1-2b2c-4eec-b3e2-85593b021123%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/12e46524-3f8f-4a1e-90d7-5ae4f4c3a191%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.