Concurrent search request to elasticsearch

Elastic Stack Elasticsearch
1

What is the maximum limit on the concurrent search requests with default Elastic search server settings.

I am able to perform only 5 parallel search requests in my application with default settings.

how can we improve the scalability of ES server search requests apart from increasing number of node,shards and queue size in search thread pool.

thanks.

2

Hello,

The search threadpool size (that is, how many requests can be actually
worked on at once) defaults to 3 times the number of processors. This might
be reduced in future, though, see:

The queue size (how many requests ES can accept before starting to reject
them) defaults to 1000.

From what I understand, this is per thread, so the answer to your question
depends on how many processors you have and how many shards get hit by each
search. For example, if a search runs on 3 indices, each with 2 shards
(number of replicas is irrelevant, because the search will only hit one
complete set of data) you'll get 6 requests in the threadpool per search.
If you have two servers with 8 cores each, you 832=48 threads available.
So the cluster can work on 8 requests at once. On top of that it can still
queue round-down(2 nodes * 1000 queue size/6 requests per search)=333
searches until it starts rejecting them.

Regarding your scaling question, I can't give you a direct answer,
unfortunately, because it depends on a whole lot of variables, mainly how
your data, queries and hardware look like and what can be changed. The fact
that your threadpool queue got full is just a symptom, it's not clear to me
what happens in there. I usually see this when there are lots of indices
and/or those indices have lots of shards. So a single request takes a lot
of requests in the threadpool, filling it up, even if the ES cluster can
keep up with the load. If that's your case increase the threadpool queue
size and make sure you don't have too many shards per index.

If your cluster can't keep up with the load (a monitoring tool like SPM
http://sematext.com/spm/ should show you that), then the first step is to
see where is the bottleneck. Again, monitoring can give some insight: are
queries too expensive, can they be optimized? do you have too many cache
evictions? is the heap size too large or too small? is memory, I/O or CPU
the bottleneck? Things like that. It could also be that you need
more/different hardware.

Finally, you can make scaling ES someone else's problem by using a hosted
service like Logsene http://sematext.com/logsene/index.html. Especially
if you're using ES for log- or metric-like data, you'll get lots of
features out of the box, and we expose most of the ES API to plug in your
custom stuff.

Best regards,
Radu

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Jan 8, 2015 at 1:32 PM, vipins sumitharjunan@gmail.com wrote:

What is the maximum limit on the concurrent search requests with default
Elastic search server settings.

I am able to perform only 5 parallel search requests in my application with
default settings.

how can we improve the scalability of ES server search requests apart from
increasing number of node,shards and queue size in search thread pool.

thanks.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1420716748150-4068702.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_2ZoQ6bewJ_aSxFMxAO0_5Mdsu%3D9WA4m_be7AfSb%3D2%2BTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks a lot for your detailed response.
We have got all default settings only.Single node and 5 shards. But there are lot of indices with huge number of records.
search settings:
"threads" : 12,
"queuesize" : 1000,

My query is very simple. which runs on a single index only.

Even with 5 requests in between it is throwing None of the configured nodes are available.

Thanks,

4

You're welcome.

So you're saying you're running 5 searches on a single index with 5 shards
(25 per-shard queries in total) and you're getting an error? I assume that
error doesn't say the queue is full because the queue is 1000. Can you post
the full error and also a gist where you reproduce the issue
http://www.elasticsearch.org/help? I might be missing an essential bit
here.

Best regards,
Radu

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Jan 8, 2015 at 3:14 PM, vipins sumitharjunan@gmail.com wrote:

Thanks a lot for your detailed response.
We have got all default settings only.Single node and 5 shards. But there
are lot of indices with huge number of records.
search settings:
"threads" : 12,
"queuesize" : 1000,

My query is very simple. which runs on a single index only.

Even with 5 requests in between it is throwing None of the configured nodes
are available.

Thanks,

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702p4068707.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1420722888084-4068707.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_0DMNN91Xo_iQpFFdep%3DosynTs5tGr-UDX_EOCOX%3DTg5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Sorry , I was wrong with number of shards. actual number of shards is 320 for the index which i am querying.

We are using rolling indices on a daily basis.

max queue size is 1000 for search thread pool.

We overcome the issue None of the configured nodes are available by keeping tcp connection alive as true.

6

OK, now it makes sense. 5 requests with 320 shards might saturate your
queue.

But 320 shards sounds like a lot for one index. I assume you don't need to
scale that very index to 320 nodes (+ replicas). If you can get the number
of shards down (say, to the default of 5) things will surely look better
not only from the queue's perspective, but it should also improve search
performance.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thu, Jan 8, 2015 at 3:46 PM, vipins sumitharjunan@gmail.com wrote:

Sorry , I was wrong with number of shards. actual number of shards is 320
for
the index which i am querying.

We are using rolling indices on a daily basis.

max queue size is 1000 for search thread pool.

We overcome the issue None of the configured nodes are available by keeping
tcp connection alive as true.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702p4068711.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1420724779413-4068711.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_00wG%2BNUQQm2_KtH7jKBC7ovN1AXnAf9Jot2VCTppMk9g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

Thanks for your prompt response.

Surely will reduce the number of shards with nodes/replicas addition for the better performance of the search.