I have a setup where there is occasional heavy indexing but it is critical
to continue serving read requests. Would it then be a good idea to assign
only n-1 cores to index and bulk threadpools (n-1 threadpool size) while
leave one core free for just search?
Unfortunately it would not be enough since there are also merging threads
that take CPI and I/O. I would recommend to stick with the defaults and
only try to change settings if you observe latency spikes at search time
which are due to indexing.
I have a setup where there is occasional heavy indexing but it is critical
to continue serving read requests. Would it then be a good idea to assign
only n-1 cores to index and bulk threadpools (n-1 threadpool size) while
leave one core free for just search?
I do observe latency spikes, and massive cluster outages during bulk
indexing. I have moved to giving only 1 thread each to bulk and index
pools. It is now a 4 core machine so I guess I have 2 cores free for search.
What are the other thread pools that can misbehave? Refresh, merge?
On Tue, Apr 14, 2015 at 7:37 PM, Adrien Grand adrien@elastic.co wrote:
Unfortunately it would not be enough since there are also merging threads
that take CPI and I/O. I would recommend to stick with the defaults and
only try to change settings if you observe latency spikes at search time
which are due to indexing.
I have a setup where there is occasional heavy indexing but it is
critical to continue serving read requests. Would it then be a good idea to
assign only n-1 cores to index and bulk threadpools (n-1 threadpool size)
while leave one core free for just search?
Yes, merges can hurt, but you can throttle
them: http://search-lucene.com/?q=throttle+merge&fc_project=ElasticSearch
You can easily correlate search latency with merges, flushes, and refreshes
with something like SPM for Elasticsearch. This could help you figure out
how much you need to throttle merges.
On Wednesday, April 15, 2015 at 1:10:04 AM UTC-4, Debayan Banerjee wrote:
I do observe latency spikes, and massive cluster outages during bulk
indexing. I have moved to giving only 1 thread each to bulk and index
pools. It is now a 4 core machine so I guess I have 2 cores free for search.
What are the other thread pools that can misbehave? Refresh, merge?
On Tue, Apr 14, 2015 at 7:37 PM, Adrien Grand <adr...@elastic.co
<javascript:>> wrote:
Unfortunately it would not be enough since there are also merging threads
that take CPI and I/O. I would recommend to stick with the defaults and
only try to change settings if you observe latency spikes at search time
which are due to indexing.
On Tue, Apr 14, 2015 at 3:57 AM, Debayan Banerjee <debayan....@paytm.com
<javascript:>> wrote:
I have a setup where there is occasional heavy indexing but it is
critical to continue serving read requests. Would it then be a good idea to
assign only n-1 cores to index and bulk threadpools (n-1 threadpool size)
while leave one core free for just search?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.