Configuring ES to reject > N concurrent requests


(Otis Gospodnetić) #1

Hi,

Is there anything in ES that could be used to control how many concurrent
connections/request each node in the cluster can handle?

I'm asking because on one largish ES system we are working on we regularly
get into a situation where for some reason queries start taking longer to
execute and ES can't keep up with search request rate. But the apps
sending search requests don't know that because ES keeps accepting search
requests, so they just keep sending more search requests to ES. Eventually
things break - the JVM becomes unresponsive, the wrapper restarts ES,
restarted nodes disconnect and reconnect, and manual intervention becomes
needed.

If we could configure each node to accept a maximum of N requests/second or
to allow for a maximum of N concurrent queries and return, say, 503 HTTP
code when this limit is reached, then it would be up to the search app to
figure out what to do and ES/JVM wouldn't fall over.

Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html


(Shay Banon) #2

You can control that by configuring the thread pool on each node. Its not
cluster wide, but it should solve what you are after. Currently, 503 is not
send as a status code though, that would be a nice addition, open an issue
for that?

On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

Is there anything in ES that could be used to control how many concurrent
connections/request each node in the cluster can handle?

I'm asking because on one largish ES system we are working on we regularly
get into a situation where for some reason queries start taking longer to
execute and ES can't keep up with search request rate. But the apps
sending search requests don't know that because ES keeps accepting search
requests, so they just keep sending more search requests to ES. Eventually
things break - the JVM becomes unresponsive, the wrapper restarts ES,
restarted nodes disconnect and reconnect, and manual intervention becomes
needed.

If we could configure each node to accept a maximum of N requests/second
or to allow for a maximum of N concurrent queries and return, say, 503 HTTP
code when this limit is reached, then it would be up to the search app to
figure out what to do and ES/JVM wouldn't fall over.

Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html


(Otis Gospodnetić) #3

Hi,

Yeah, found the threadpool stuff in the docs and set the queue for the
search threadpool to fixed with queue size of 0 and rejection policy set to
caller:

threadpool:
search:
type: fixed
size: 120
queue: 0
reject_policy: caller

However, after some ES hammering I see this:

"search" : {
"threads" : 500, // set to 120 in the config
"queue" : 5795, // set to 0 in the config
"active" : 500
}

My hope was that by setting the queue size to 0 ES/Netty would start
rejecting connections after the limit of 120 concurrent connections has
been reached.
Also, my understanding was that by setting "size" to 120 I would instruct
ES not to allow more than 120 concurrent connections. Since I see 500
(active) threads up there, I'm wondering if "size" doesn't actually control
that setting?

This is ES 0.19.3.

I'll open the issue for 503 now.

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote:

You can control that by configuring the thread pool on each node. Its not
cluster wide, but it should solve what you are after. Currently, 503 is not
send as a status code though, that would be a nice addition, open an issue
for that?

On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

Is there anything in ES that could be used to control how many concurrent
connections/request each node in the cluster can handle?

I'm asking because on one largish ES system we are working on we
regularly get into a situation where for some reason queries start taking
longer to execute and ES can't keep up with search request rate. But the
apps sending search requests don't know that because ES keeps accepting
search requests, so they just keep sending more search requests to ES.
Eventually things break - the JVM becomes unresponsive, the wrapper
restarts ES, restarted nodes disconnect and reconnect, and manual
intervention becomes needed.

If we could configure each node to accept a maximum of N requests/second
or to allow for a maximum of N concurrent queries and return, say, 503 HTTP
code when this limit is reached, then it would be up to the search app to
figure out what to do and ES/JVM wouldn't fall over.

Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html


(Otis Gospodnetić) #4

Correction. I "lied" about threads == 500 when size was set to 120.

But I didn't lie about the queue being non-zero when "queue" in the config
was set to 0 as shown below.

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Friday, June 22, 2012 4:04:06 PM UTC-4, Otis Gospodnetic wrote:

Hi,

Yeah, found the threadpool stuff in the docs and set the queue for the
search threadpool to fixed with queue size of 0 and rejection policy set to
caller:

threadpool:
search:
type: fixed
size: 120
queue: 0
reject_policy: caller

However, after some ES hammering I see this:

"search" : {
"threads" : 500, // set to 120 in the config
"queue" : 5795, // set to 0 in the config
"active" : 500
}

My hope was that by setting the queue size to 0 ES/Netty would start
rejecting connections after the limit of 120 concurrent connections has
been reached.
Also, my understanding was that by setting "size" to 120 I would instruct
ES not to allow more than 120 concurrent connections. Since I see 500
(active) threads up there, I'm wondering if "size" doesn't actually control
that setting?

This is ES 0.19.3.

I'll open the issue for 503 now.

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote:

You can control that by configuring the thread pool on each node. Its not
cluster wide, but it should solve what you are after. Currently, 503 is not
send as a status code though, that would be a nice addition, open an issue
for that?

On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

Is there anything in ES that could be used to control how many
concurrent connections/request each node in the cluster can handle?

I'm asking because on one largish ES system we are working on we
regularly get into a situation where for some reason queries start taking
longer to execute and ES can't keep up with search request rate. But the
apps sending search requests don't know that because ES keeps accepting
search requests, so they just keep sending more search requests to ES.
Eventually things break - the JVM becomes unresponsive, the wrapper
restarts ES, restarted nodes disconnect and reconnect, and manual
intervention becomes needed.

If we could configure each node to accept a maximum of N requests/second
or to allow for a maximum of N concurrent queries and return, say, 503 HTTP
code when this limit is reached, then it would be up to the search app to
figure out what to do and ES/JVM wouldn't fall over.

Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html


(Shay Banon) #5

Heya,

The setting for the queue size is queue_size, not queue :). Also,
reject_policy set to caller is probably not what you want, but the default
abort one is the one you should use, otherwise, it will still be executed,
just on the socket reading thread in this case.

On Fri, Jun 22, 2012 at 10:10 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Correction. I "lied" about threads == 500 when size was set to 120.

But I didn't lie about the queue being non-zero when "queue" in the config
was set to 0 as shown below.

Otis

Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.**htmlhttp://sematext.com/spm/index.html

On Friday, June 22, 2012 4:04:06 PM UTC-4, Otis Gospodnetic wrote:

Hi,

Yeah, found the threadpool stuff in the docs and set the queue for the
search threadpool to fixed with queue size of 0 and rejection policy set to
caller:

threadpool:
search:
type: fixed
size: 120
queue: 0
reject_policy: caller

However, after some ES hammering I see this:

"search" : {
"threads" : 500, // set to 120 in the config
"queue" : 5795, // set to 0 in the config
"active" : 500
}

My hope was that by setting the queue size to 0 ES/Netty would start
rejecting connections after the limit of 120 concurrent connections has
been reached.
Also, my understanding was that by setting "size" to 120 I would instruct
ES not to allow more than 120 concurrent connections. Since I see 500
(active) threads up there, I'm wondering if "size" doesn't actually control
that setting?

This is ES 0.19.3.

I'll open the issue for 503 now.

Thanks,
Otis

Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.**htmlhttp://sematext.com/spm/index.html

On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote:

You can control that by configuring the thread pool on each node. Its
not cluster wide, but it should solve what you are after. Currently, 503 is
not send as a status code though, that would be a nice addition, open an
issue for that?

On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

Is there anything in ES that could be used to control how many
concurrent connections/request each node in the cluster can handle?

I'm asking because on one largish ES system we are working on we
regularly get into a situation where for some reason queries start taking
longer to execute and ES can't keep up with search request rate. But the
apps sending search requests don't know that because ES keeps accepting
search requests, so they just keep sending more search requests to ES.
Eventually things break - the JVM becomes unresponsive, the wrapper
restarts ES, restarted nodes disconnect and reconnect, and manual
intervention becomes needed.

If we could configure each node to accept a maximum of N
requests/second or to allow for a maximum of N concurrent queries and
return, say, 503 HTTP code when this limit is reached, then it would be up
to the search app to figure out what to do and ES/JVM wouldn't fall over.

Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?

Thanks,
Otis

Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.**htmlhttp://sematext.com/spm/index.html


(Otis Gospodnetić) #6

Hi Shay,

Aaah, thanks! Note the docs on
http://www.elasticsearch.org/guide/reference/modules/threadpool.html - a
bit confusing to have "queue_size" for setting the queue size and then just
"queue" for showing the queue size.

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Monday, June 25, 2012 8:29:46 AM UTC-4, kimchy wrote:

Heya,

The setting for the queue size is queue_size, not queue :). Also,
reject_policy set to caller is probably not what you want, but the default
abort one is the one you should use, otherwise, it will still be executed,
just on the socket reading thread in this case.

On Fri, Jun 22, 2012 at 10:10 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Correction. I "lied" about threads == 500 when size was set to 120.

But I didn't lie about the queue being non-zero when "queue" in the
config was set to 0 as shown below.

Otis

Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.**htmlhttp://sematext.com/spm/index.html

On Friday, June 22, 2012 4:04:06 PM UTC-4, Otis Gospodnetic wrote:

Hi,

Yeah, found the threadpool stuff in the docs and set the queue for the
search threadpool to fixed with queue size of 0 and rejection policy set to
caller:

threadpool:
search:
type: fixed
size: 120
queue: 0
reject_policy: caller

However, after some ES hammering I see this:

"search" : {
"threads" : 500, // set to 120 in the config
"queue" : 5795, // set to 0 in the config
"active" : 500
}

My hope was that by setting the queue size to 0 ES/Netty would start
rejecting connections after the limit of 120 concurrent connections has
been reached.
Also, my understanding was that by setting "size" to 120 I would
instruct ES not to allow more than 120 concurrent connections. Since I see
500 (active) threads up there, I'm wondering if "size" doesn't actually
control that setting?

This is ES 0.19.3.

I'll open the issue for 503 now.

Thanks,
Otis

Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.**htmlhttp://sematext.com/spm/index.html

On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote:

You can control that by configuring the thread pool on each node. Its
not cluster wide, but it should solve what you are after. Currently, 503 is
not send as a status code though, that would be a nice addition, open an
issue for that?

On Wed, Jun 20, 2012 at 10:35 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

Is there anything in ES that could be used to control how many
concurrent connections/request each node in the cluster can handle?

I'm asking because on one largish ES system we are working on we
regularly get into a situation where for some reason queries start taking
longer to execute and ES can't keep up with search request rate. But the
apps sending search requests don't know that because ES keeps accepting
search requests, so they just keep sending more search requests to ES.
Eventually things break - the JVM becomes unresponsive, the wrapper
restarts ES, restarted nodes disconnect and reconnect, and manual
intervention becomes needed.

If we could configure each node to accept a maximum of N
requests/second or to allow for a maximum of N concurrent queries and
return, say, 503 HTTP code when this limit is reached, then it would be up
to the search app to figure out what to do and ES/JVM wouldn't fall over.

Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?

Thanks,
Otis

Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.**htmlhttp://sematext.com/spm/index.html


(system) #7