Is there anything in ES that could be used to control how many concurrent
connections/request each node in the cluster can handle?
I'm asking because on one largish ES system we are working on we regularly
get into a situation where for some reason queries start taking longer to
execute and ES can't keep up with search request rate. But the apps
sending search requests don't know that because ES keeps accepting search
requests, so they just keep sending more search requests to ES. Eventually
things break - the JVM becomes unresponsive, the wrapper restarts ES,
restarted nodes disconnect and reconnect, and manual intervention becomes
needed.
If we could configure each node to accept a maximum of N requests/second or
to allow for a maximum of N concurrent queries and return, say, 503 HTTP
code when this limit is reached, then it would be up to the search app to
figure out what to do and ES/JVM wouldn't fall over.
Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?
You can control that by configuring the thread pool on each node. Its not
cluster wide, but it should solve what you are after. Currently, 503 is not
send as a status code though, that would be a nice addition, open an issue
for that?
Is there anything in ES that could be used to control how many concurrent
connections/request each node in the cluster can handle?
I'm asking because on one largish ES system we are working on we regularly
get into a situation where for some reason queries start taking longer to
execute and ES can't keep up with search request rate. But the apps
sending search requests don't know that because ES keeps accepting search
requests, so they just keep sending more search requests to ES. Eventually
things break - the JVM becomes unresponsive, the wrapper restarts ES,
restarted nodes disconnect and reconnect, and manual intervention becomes
needed.
If we could configure each node to accept a maximum of N requests/second
or to allow for a maximum of N concurrent queries and return, say, 503 HTTP
code when this limit is reached, then it would be up to the search app to
figure out what to do and ES/JVM wouldn't fall over.
Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?
Yeah, found the threadpool stuff in the docs and set the queue for the
search threadpool to fixed with queue size of 0 and rejection policy set to
caller:
"search" : {
"threads" : 500, // set to 120 in the config
"queue" : 5795, // set to 0 in the config
"active" : 500
}
My hope was that by setting the queue size to 0 ES/Netty would start
rejecting connections after the limit of 120 concurrent connections has
been reached.
Also, my understanding was that by setting "size" to 120 I would instruct
ES not to allow more than 120 concurrent connections. Since I see 500
(active) threads up there, I'm wondering if "size" doesn't actually control
that setting?
On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote:
You can control that by configuring the thread pool on each node. Its not
cluster wide, but it should solve what you are after. Currently, 503 is not
send as a status code though, that would be a nice addition, open an issue
for that?
Is there anything in ES that could be used to control how many concurrent
connections/request each node in the cluster can handle?
I'm asking because on one largish ES system we are working on we
regularly get into a situation where for some reason queries start taking
longer to execute and ES can't keep up with search request rate. But the
apps sending search requests don't know that because ES keeps accepting
search requests, so they just keep sending more search requests to ES.
Eventually things break - the JVM becomes unresponsive, the wrapper
restarts ES, restarted nodes disconnect and reconnect, and manual
intervention becomes needed.
If we could configure each node to accept a maximum of N requests/second
or to allow for a maximum of N concurrent queries and return, say, 503 HTTP
code when this limit is reached, then it would be up to the search app to
figure out what to do and ES/JVM wouldn't fall over.
Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?
On Friday, June 22, 2012 4:04:06 PM UTC-4, Otis Gospodnetic wrote:
Hi,
Yeah, found the threadpool stuff in the docs and set the queue for the
search threadpool to fixed with queue size of 0 and rejection policy set to
caller:
"search" : {
"threads" : 500, // set to 120 in the config
"queue" : 5795, // set to 0 in the config
"active" : 500
}
My hope was that by setting the queue size to 0 ES/Netty would start
rejecting connections after the limit of 120 concurrent connections has
been reached.
Also, my understanding was that by setting "size" to 120 I would instruct
ES not to allow more than 120 concurrent connections. Since I see 500
(active) threads up there, I'm wondering if "size" doesn't actually control
that setting?
On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote:
You can control that by configuring the thread pool on each node. Its not
cluster wide, but it should solve what you are after. Currently, 503 is not
send as a status code though, that would be a nice addition, open an issue
for that?
Is there anything in ES that could be used to control how many
concurrent connections/request each node in the cluster can handle?
I'm asking because on one largish ES system we are working on we
regularly get into a situation where for some reason queries start taking
longer to execute and ES can't keep up with search request rate. But the
apps sending search requests don't know that because ES keeps accepting
search requests, so they just keep sending more search requests to ES.
Eventually things break - the JVM becomes unresponsive, the wrapper
restarts ES, restarted nodes disconnect and reconnect, and manual
intervention becomes needed.
If we could configure each node to accept a maximum of N requests/second
or to allow for a maximum of N concurrent queries and return, say, 503 HTTP
code when this limit is reached, then it would be up to the search app to
figure out what to do and ES/JVM wouldn't fall over.
Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?
The setting for the queue size is queue_size, not queue :). Also,
reject_policy set to caller is probably not what you want, but the default
abort one is the one you should use, otherwise, it will still be executed,
just on the socket reading thread in this case.
On Friday, June 22, 2012 4:04:06 PM UTC-4, Otis Gospodnetic wrote:
Hi,
Yeah, found the threadpool stuff in the docs and set the queue for the
search threadpool to fixed with queue size of 0 and rejection policy set to
caller:
"search" : {
"threads" : 500, // set to 120 in the config
"queue" : 5795, // set to 0 in the config
"active" : 500
}
My hope was that by setting the queue size to 0 ES/Netty would start
rejecting connections after the limit of 120 concurrent connections has
been reached.
Also, my understanding was that by setting "size" to 120 I would instruct
ES not to allow more than 120 concurrent connections. Since I see 500
(active) threads up there, I'm wondering if "size" doesn't actually control
that setting?
On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote:
You can control that by configuring the thread pool on each node. Its
not cluster wide, but it should solve what you are after. Currently, 503 is
not send as a status code though, that would be a nice addition, open an
issue for that?
Is there anything in ES that could be used to control how many
concurrent connections/request each node in the cluster can handle?
I'm asking because on one largish ES system we are working on we
regularly get into a situation where for some reason queries start taking
longer to execute and ES can't keep up with search request rate. But the
apps sending search requests don't know that because ES keeps accepting
search requests, so they just keep sending more search requests to ES.
Eventually things break - the JVM becomes unresponsive, the wrapper
restarts ES, restarted nodes disconnect and reconnect, and manual
intervention becomes needed.
If we could configure each node to accept a maximum of N
requests/second or to allow for a maximum of N concurrent queries and
return, say, 503 HTTP code when this limit is reached, then it would be up
to the search app to figure out what to do and ES/JVM wouldn't fall over.
Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?
On Monday, June 25, 2012 8:29:46 AM UTC-4, kimchy wrote:
Heya,
The setting for the queue size is queue_size, not queue :). Also,
reject_policy set to caller is probably not what you want, but the default
abort one is the one you should use, otherwise, it will still be executed,
just on the socket reading thread in this case.
On Friday, June 22, 2012 4:04:06 PM UTC-4, Otis Gospodnetic wrote:
Hi,
Yeah, found the threadpool stuff in the docs and set the queue for the
search threadpool to fixed with queue size of 0 and rejection policy set to
caller:
"search" : {
"threads" : 500, // set to 120 in the config
"queue" : 5795, // set to 0 in the config
"active" : 500
}
My hope was that by setting the queue size to 0 ES/Netty would start
rejecting connections after the limit of 120 concurrent connections has
been reached.
Also, my understanding was that by setting "size" to 120 I would
instruct ES not to allow more than 120 concurrent connections. Since I see
500 (active) threads up there, I'm wondering if "size" doesn't actually
control that setting?
On Thursday, June 21, 2012 3:21:36 AM UTC-4, kimchy wrote:
You can control that by configuring the thread pool on each node. Its
not cluster wide, but it should solve what you are after. Currently, 503 is
not send as a status code though, that would be a nice addition, open an
issue for that?
Is there anything in ES that could be used to control how many
concurrent connections/request each node in the cluster can handle?
I'm asking because on one largish ES system we are working on we
regularly get into a situation where for some reason queries start taking
longer to execute and ES can't keep up with search request rate. But the
apps sending search requests don't know that because ES keeps accepting
search requests, so they just keep sending more search requests to ES.
Eventually things break - the JVM becomes unresponsive, the wrapper
restarts ES, restarted nodes disconnect and reconnect, and manual
intervention becomes needed.
If we could configure each node to accept a maximum of N
requests/second or to allow for a maximum of N concurrent queries and
return, say, 503 HTTP code when this limit is reached, then it would be up
to the search app to figure out what to do and ES/JVM wouldn't fall over.
Is this currently doable? I couldn't find it in the docs.
If not, should I open an issue?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.