# Configuring ES to reject \> N concurrent requests **URL:** \ **Category:** Elasticsearch\ **Created:** [June 20, 2012, 8:35pm UTC](https://discuss.elastic.co/t/configuring-es-to-reject-n-concurrent-requests/8175 "2012-06-20T20:35:54Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 20, 2012, 8:35pm UTC](https://discuss.elastic.co/t/configuring-es-to-reject-n-concurrent-requests/8175/1 "2012-06-20T20:35:54Z")
Hi, Is there anything in ES that could be used to control how many concurrent connections/request each node in the cluster can handle? I'm asking because on one largish ES system we are working on we regularly get into a situation where for some reason queries start taking longer to execute and ES can't keep up with search request rate. But the apps sending search requests don't know that because ES keeps accepting search requests, so they just keep sending more search requests to ES. Eventually things break - the JVM becomes unresponsive, the wrapper restarts ES, restarted nodes disconnect and reconnect, and manual intervention becomes needed. If we could configure each node to accept a maximum of N requests/second or to allow for a maximum of N concurrent queries and return, say, 503 HTTP code when this limit is reached, then it would be up to the search app to figure out what to do and ES/JVM wouldn't fall over. Is this currently doable? I couldn't find it in the docs. If not, should I open an issue? ## Thanks, Otis Search Analytics - [http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 21, 2012, 7:21am UTC](https://discuss.elastic.co/t/configuring-es-to-reject-n-concurrent-requests/8175/2 "2012-06-21T07:21:36Z")
You can control that by configuring the thread pool on each node. Its not cluster wide, but it should solve what you are after. Currently, 503 is not send as a status code though, that would be a nice addition, open an issue for that? On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic \< [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > Hi, > > Is there anything in ES that could be used to control how many concurrent > connections/request each node in the cluster can handle? > > I'm asking because on one largish ES system we are working on we regularly > get into a situation where for some reason queries start taking longer to > execute and ES can't keep up with search request rate. But the apps > sending search requests don't know that because ES keeps accepting search > requests, so they just keep sending more search requests to ES. Eventually > things break - the JVM becomes unresponsive, the wrapper restarts ES, > restarted nodes disconnect and reconnect, and manual intervention becomes > needed. > > If we could configure each node to accept a maximum of N requests/second > or to allow for a maximum of N concurrent queries and return, say, 503 HTTP > code when this limit is reached, then it would be up to the search app to > figure out what to do and ES/JVM wouldn't fall over. > > Is this currently doable? I couldn't find it in the docs. > If not, should I open an issue? > > ## Thanks, Otis > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 22, 2012, 8:04pm UTC](https://discuss.elastic.co/t/configuring-es-to-reject-n-concurrent-requests/8175/3 "2012-06-22T20:04:06Z")
Hi, Yeah, found the threadpool stuff in the docs and set the queue for the search threadpool to fixed with queue size of 0 and rejection policy set to caller: threadpool: search: type: fixed size: 120 queue: 0 reject\_policy: caller However, after some ES hammering I see this: "search" : { "threads" : 500, // set to 120 in the config "queue" : 5795, // set to 0 in the config "active" : 500 } My hope was that by setting the queue size to 0 ES/Netty would start rejecting connections after the limit of 120 concurrent connections has been reached. Also, my understanding was that by setting "size" to 120 I would instruct ES not to allow more than 120 concurrent connections. Since I see 500 (active) threads up there, I'm wondering if "size" doesn't actually control that setting? This is ES 0.19.3. I'll open the issue for 503 now. ## Thanks, Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote: > You can control that by configuring the thread pool on each node. Its not > cluster wide, but it should solve what you are after. Currently, 503 is not > send as a status code though, that would be a nice addition, open an issue > for that? > > On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic \< > [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > > > Hi, > > > > Is there anything in ES that could be used to control how many concurrent > > connections/request each node in the cluster can handle? > > > > I'm asking because on one largish ES system we are working on we > > regularly get into a situation where for some reason queries start taking > > longer to execute and ES can't keep up with search request rate. But the > > apps sending search requests don't know that because ES keeps accepting > > search requests, so they just keep sending more search requests to ES. > > Eventually things break - the JVM becomes unresponsive, the wrapper > > restarts ES, restarted nodes disconnect and reconnect, and manual > > intervention becomes needed. > > > > If we could configure each node to accept a maximum of N requests/second > > or to allow for a maximum of N concurrent queries and return, say, 503 HTTP > > code when this limit is reached, then it would be up to the search app to > > figure out what to do and ES/JVM wouldn't fall over. > > > > Is this currently doable? I couldn't find it in the docs. > > If not, should I open an issue? > > > > ## Thanks, Otis > > > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > > Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 22, 2012, 8:10pm UTC](https://discuss.elastic.co/t/configuring-es-to-reject-n-concurrent-requests/8175/4 "2012-06-22T20:10:50Z")
Correction. I "lied" about threads == 500 when size was set to 120. But I didn't lie about the queue being non-zero when "queue" in the config was set to 0 as shown below. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Friday, June 22, 2012 4:04:06 PM UTC-4, Otis Gospodnetic wrote: > Hi, > > Yeah, found the threadpool stuff in the docs and set the queue for the > search threadpool to fixed with queue size of 0 and rejection policy set to > caller: > > threadpool: > search: > type: fixed > size: 120 > queue: 0 > reject\_policy: caller > > However, after some ES hammering I see this: > > "search" : { > "threads" : 500, // set to 120 in the config > "queue" : 5795, // set to 0 in the config > "active" : 500 > } > > My hope was that by setting the queue size to 0 ES/Netty would start > rejecting connections after the limit of 120 concurrent connections has > been reached. > Also, my understanding was that by setting "size" to 120 I would instruct > ES not to allow more than 120 concurrent connections. Since I see 500 > (active) threads up there, I'm wondering if "size" doesn't actually control > that setting? > > This is ES 0.19.3. > > I'll open the issue for 503 now. > > ## Thanks, Otis > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote: > > > You can control that by configuring the thread pool on each node. Its not > > cluster wide, but it should solve what you are after. Currently, 503 is not > > send as a status code though, that would be a nice addition, open an issue > > for that? > > > > On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic \< > > [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > > > > > Hi, > > > > > > Is there anything in ES that could be used to control how many > > > concurrent connections/request each node in the cluster can handle? > > > > > > I'm asking because on one largish ES system we are working on we > > > regularly get into a situation where for some reason queries start taking > > > longer to execute and ES can't keep up with search request rate. But the > > > apps sending search requests don't know that because ES keeps accepting > > > search requests, so they just keep sending more search requests to ES. > > > Eventually things break - the JVM becomes unresponsive, the wrapper > > > restarts ES, restarted nodes disconnect and reconnect, and manual > > > intervention becomes needed. > > > > > > If we could configure each node to accept a maximum of N requests/second > > > or to allow for a maximum of N concurrent queries and return, say, 503 HTTP > > > code when this limit is reached, then it would be up to the search app to > > > figure out what to do and ES/JVM wouldn't fall over. > > > > > > Is this currently doable? I couldn't find it in the docs. > > > If not, should I open an issue? > > > > > > ## Thanks, Otis > > > > > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > > > Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 25, 2012, 12:29pm UTC](https://discuss.elastic.co/t/configuring-es-to-reject-n-concurrent-requests/8175/5 "2012-06-25T12:29:46Z")
Heya, The setting for the queue size is queue\_size, not queue :). Also, reject\_policy set to caller is probably not what you want, but the default abort one is the one you should use, otherwise, it will still be executed, just on the socket reading thread in this case. On Fri, Jun 22, 2012 at 10:10 PM, Otis Gospodnetic \< [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > Correction. I "lied" about threads == 500 when size was set to 120. > > But I didn't lie about the queue being non-zero when "queue" in the config > was set to 0 as shown below. > > ## Otis > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > Scalable Performance Monitoring - [http://sematext.com/spm/index.\*\*html](http://sematext.com/spm/index.**html)[http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > On Friday, June 22, 2012 4:04:06 PM UTC-4, Otis Gospodnetic wrote: > > > Hi, > > > > Yeah, found the threadpool stuff in the docs and set the queue for the > > search threadpool to fixed with queue size of 0 and rejection policy set to > > caller: > > > > threadpool: > > search: > > type: fixed > > size: 120 > > queue: 0 > > reject\_policy: caller > > > > However, after some ES hammering I see this: > > > > "search" : { > > "threads" : 500, // set to 120 in the config > > "queue" : 5795, // set to 0 in the config > > "active" : 500 > > } > > > > My hope was that by setting the queue size to 0 ES/Netty would start > > rejecting connections after the limit of 120 concurrent connections has > > been reached. > > Also, my understanding was that by setting "size" to 120 I would instruct > > ES not to allow more than 120 concurrent connections. Since I see 500 > > (active) threads up there, I'm wondering if "size" doesn't actually control > > that setting? > > > > This is ES 0.19.3. > > > > I'll open the issue for 503 now. > > > > ## Thanks, Otis > > > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > Scalable Performance Monitoring - [http://sematext.com/spm/index.\*\*html](http://sematext.com/spm/index.**html)[http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote: > > > > > You can control that by configuring the thread pool on each node. Its > > > not cluster wide, but it should solve what you are after. Currently, 503 is > > > not send as a status code though, that would be a nice addition, open an > > > issue for that? > > > > > > On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic \< > > > [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > > > > > > > Hi, > > > > > > > > Is there anything in ES that could be used to control how many > > > > concurrent connections/request each node in the cluster can handle? > > > > > > > > I'm asking because on one largish ES system we are working on we > > > > regularly get into a situation where for some reason queries start taking > > > > longer to execute and ES can't keep up with search request rate. But the > > > > apps sending search requests don't know that because ES keeps accepting > > > > search requests, so they just keep sending more search requests to ES. > > > > Eventually things break - the JVM becomes unresponsive, the wrapper > > > > restarts ES, restarted nodes disconnect and reconnect, and manual > > > > intervention becomes needed. > > > > > > > > If we could configure each node to accept a maximum of N > > > > requests/second or to allow for a maximum of N concurrent queries and > > > > return, say, 503 HTTP code when this limit is reached, then it would be up > > > > to the search app to figure out what to do and ES/JVM wouldn't fall over. > > > > > > > > Is this currently doable? I couldn't find it in the docs. > > > > If not, should I open an issue? > > > > > > > > ## Thanks, Otis > > > > > > > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > Scalable Performance Monitoring - [http://sematext.com/spm/index.\*\*html](http://sematext.com/spm/index.**html)[http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 25, 2012, 5:41pm UTC](https://discuss.elastic.co/t/configuring-es-to-reject-n-concurrent-requests/8175/6 "2012-06-25T17:41:19Z")
Hi Shay, Aaah, thanks! Note the docs on [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/modules/threadpool.html) - a bit confusing to have "queue\_size" for setting the queue size and then just "queue" for showing the queue size. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Monday, June 25, 2012 8:29:46 AM UTC-4, kimchy wrote: > Heya, > > The setting for the queue size is queue\_size, not queue :). Also, > reject\_policy set to caller is probably not what you want, but the default > abort one is the one you should use, otherwise, it will still be executed, > just on the socket reading thread in this case. > > On Fri, Jun 22, 2012 at 10:10 PM, Otis Gospodnetic \< > [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > > > Correction. I "lied" about threads == 500 when size was set to 120. > > > > But I didn't lie about the queue being non-zero when "queue" in the > > config was set to 0 as shown below. > > > > ## Otis > > > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > Scalable Performance Monitoring - [http://sematext.com/spm/index.\*\*html](http://sematext.com/spm/index.**html)[http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > On Friday, June 22, 2012 4:04:06 PM UTC-4, Otis Gospodnetic wrote: > > > > > Hi, > > > > > > Yeah, found the threadpool stuff in the docs and set the queue for the > > > search threadpool to fixed with queue size of 0 and rejection policy set to > > > caller: > > > > > > threadpool: > > > search: > > > type: fixed > > > size: 120 > > > queue: 0 > > > reject\_policy: caller > > > > > > However, after some ES hammering I see this: > > > > > > "search" : { > > > "threads" : 500, // set to 120 in the config > > > "queue" : 5795, // set to 0 in the config > > > "active" : 500 > > > } > > > > > > My hope was that by setting the queue size to 0 ES/Netty would start > > > rejecting connections after the limit of 120 concurrent connections has > > > been reached. > > > Also, my understanding was that by setting "size" to 120 I would > > > instruct ES not to allow more than 120 concurrent connections. Since I see > > > 500 (active) threads up there, I'm wondering if "size" doesn't actually > > > control that setting? > > > > > > This is ES 0.19.3. > > > > > > I'll open the issue for 503 now. > > > > > > ## Thanks, Otis > > > > > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > Scalable Performance Monitoring - [http://sematext.com/spm/index.\*\*html](http://sematext.com/spm/index.**html)[http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > > > On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote: > > > > > > > You can control that by configuring the thread pool on each node. Its > > > > not cluster wide, but it should solve what you are after. Currently, 503 is > > > > not send as a status code though, that would be a nice addition, open an > > > > issue for that? > > > > > > > > On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic \< > > > > [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > > > > > > > > > Hi, > > > > > > > > > > Is there anything in ES that could be used to control how many > > > > > concurrent connections/request each node in the cluster can handle? > > > > > > > > > > I'm asking because on one largish ES system we are working on we > > > > > regularly get into a situation where for some reason queries start taking > > > > > longer to execute and ES can't keep up with search request rate. But the > > > > > apps sending search requests don't know that because ES keeps accepting > > > > > search requests, so they just keep sending more search requests to ES. > > > > > Eventually things break - the JVM becomes unresponsive, the wrapper > > > > > restarts ES, restarted nodes disconnect and reconnect, and manual > > > > > intervention becomes needed. > > > > > > > > > > If we could configure each node to accept a maximum of N > > > > > requests/second or to allow for a maximum of N concurrent queries and > > > > > return, say, 503 HTTP code when this limit is reached, then it would be up > > > > > to the search app to figure out what to do and ES/JVM wouldn't fall over. > > > > > > > > > > Is this currently doable? I couldn't find it in the docs. > > > > > If not, should I open an issue? > > > > > > > > > > ## Thanks, Otis > > > > > > > > > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > Scalable Performance Monitoring - [http://sematext.com/spm/index.\*\*html](http://sematext.com/spm/index.**html)[http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:22am UTC](https://discuss.elastic.co/t/configuring-es-to-reject-n-concurrent-requests/8175/7 "2017-07-06T03:22:36Z")