# Understanding Threadpools **URL:** \ **Category:** Elasticsearch\ **Created:** [January 18, 2013, 11:03pm UTC](https://discuss.elastic.co/t/understanding-threadpools/10417 "2013-01-18T23:03:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![avin](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@avin](https://discuss.elastic.co/u/avin)\ **Post date:** [January 18, 2013, 11:03pm UTC](https://discuss.elastic.co/t/understanding-threadpools/10417/1 "2013-01-18T23:03:20Z")
I found that I was hitting a wall, and no matter how much I increased my thread-count in JMeter, the throughput remained constant. I was wondering if this is because the elasticsearch node can only process a certain number. I found out that tweaking the threadpool settings in the elasticsearch.yml file also made no difference threadpool: index: type: fixed size: 100 queue\_size: 10000 reject\_policy: caller Can someone give me a few pointers here -- ---
**Author:** ![egaumer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/egaumer/32/2365_2.png) [@egaumer](https://discuss.elastic.co/u/egaumer)\ **Post date:** [January 18, 2013, 11:51pm UTC](https://discuss.elastic.co/t/understanding-threadpools/10417/2 "2013-01-18T23:51:42Z")
At some point you'll end up with contention at the shard level (blocked threads waiting to write). If you want to index faster, you need to parallelize it. Add more shards, either on the existing nodes or by adding new nodes. For a frame of reference, we ran some benchmarks on a 24 core machine. With 1 shard, we got an average throughput of ~1200 docs/sec. With 20 shards on the same machine we got ~22K docs/sec (near linear improvement). Also, if possible, turn off replication when you're doing heavy indexing. Elasticsearch is not "eventually consistent" (by default), index requests don't return until the replicas have also indexed the document. If you can't disable replication (maybe your feeds are real-time) then consider using asynchronous replication. Most folks only consider "scaling out" (i.e., adding more physical nodes) but elasticsearch scales up just the same by increasing the shard allocation per node. Of course, it helps to have a core/cpu for each shard. On Friday, January 18, 2013 6:03:20 PM UTC-5, [avins...@gmail.com](mailto:avins...@gmail.com) wrote: > I found that I was hitting a wall, and no matter how much I increased my > thread-count in JMeter, the throughput remained constant. I was wondering > if this is because the elasticsearch node can only process a certain > number. I found out that tweaking the threadpool settings in the > elasticsearch.yml file also made no difference > > threadpool: > index: > type: fixed > size: 100 > queue\_size: 10000 > reject\_policy: caller > > Can someone give me a few pointers here -- ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [January 19, 2013, 10:05am UTC](https://discuss.elastic.co/t/understanding-threadpools/10417/3 "2013-01-19T10:05:10Z")
> If you want to index faster, you need to parallelize it. Add more > shards, either on the existing nodes or by adding new nodes. For a > frame of reference, we ran some benchmarks on a 24 core machine. With > 1 shard, we got an average throughput of ~1200 docs/sec. With 20 > shards on the same machine we got ~22K docs/sec (near linear > improvement). that's interesting - I thought that 1 shard would be able use all cores on the machine. Although, I seem to remember hearing that with indexing there is less parallelism. And I also seem to remember that in Lucene 4, indexing happens more in parallel than in 3.\*. Can anybody confirm? clint -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [January 21, 2013, 3:10am UTC](https://discuss.elastic.co/t/understanding-threadpools/10417/4 "2013-01-21T03:10:56Z")
Hi Clint, Correct. Simon W. is one of those bad people responsible for faster indexing. He even gave a few presentations describing his work. Let me find one... eh, couldn't quickly find it, but I found something with a pretty graph that's worth at least a few dozen words and a couple of slides: [Changing Bits: 265% indexing speedup with Lucene's concurrent flushing](http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html) ## Otis ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Saturday, January 19, 2013 5:05:10 AM UTC-5, Clinton Gormley wrote: > > If you want to index faster, you need to parallelize it. Add more > > shards, either on the existing nodes or by adding new nodes. For a > > frame of reference, we ran some benchmarks on a 24 core machine. With > > 1 shard, we got an average throughput of ~1200 docs/sec. With 20 > > shards on the same machine we got ~22K docs/sec (near linear > > improvement). > > that's interesting - I thought that 1 shard would be able use all cores > on the machine. Although, I seem to remember hearing that with indexing > there is less parallelism. And I also seem to remember that in Lucene > 4, indexing happens more in parallel than in 3.\*. > > Can anybody confirm? > > clint -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [January 21, 2013, 3:13am UTC](https://discuss.elastic.co/t/understanding-threadpools/10417/5 "2013-01-21T03:13:53Z")
Hi, I'm guessing this has to do with serialization that goes down below IndexWriter. Going from 1 shard to 20 shards means going from 1 IndexWriter to 20 IWs. As long as there is sufficient IO and enough free CPU cycles, this has to improve the throughput. Eric - did you try going beyond 20 shards? ## Otis ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Friday, January 18, 2013 6:51:42 PM UTC-5, egaumer wrote: > At some point you'll end up with contention at the shard level (blocked > threads waiting to write). > > If you want to index faster, you need to parallelize it. Add more shards, > either on the existing nodes or by adding new nodes. For a frame of > reference, we ran some benchmarks on a 24 core machine. With 1 shard, we > got an average throughput of ~1200 docs/sec. With 20 shards on the same > machine we got ~22K docs/sec (near linear improvement). > > Also, if possible, turn off replication when you're doing heavy indexing. > Elasticsearch is not "eventually consistent" (by default), index requests > don't return until the replicas have also indexed the document. If you > can't disable replication (maybe your feeds are real-time) then consider > using asynchronous replication. > > Most folks only consider "scaling out" (i.e., adding more physical nodes) > but elasticsearch scales up just the same by increasing the shard > allocation per node. Of course, it helps to have a core/cpu for each shard. > > On Friday, January 18, 2013 6:03:20 PM UTC-5, [avins...@gmail.com](mailto:avins...@gmail.com) wrote: > > > I found that I was hitting a wall, and no matter how much I increased my > > thread-count in JMeter, the throughput remained constant. I was wondering > > if this is because the elasticsearch node can only process a certain > > number. I found out that tweaking the threadpool settings in the > > elasticsearch.yml file also made no difference > > > > threadpool: > > index: > > type: fixed > > size: 100 > > queue\_size: 10000 > > reject\_policy: caller > > > > Can someone give me a few pointers here -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 21, 2013, 4:27pm UTC](https://discuss.elastic.co/t/understanding-threadpools/10417/6 "2013-01-21T16:27:38Z")
Increasing shard count per node scales well as long as - the IO subsystem is not satisfied - the CPU can manage the load created by the increasing number of threads The slowest part of the machine is the disk subsystem (if it's SSD or spindle disk makes no big difference in that point). It means tuning the disk I/O will give the highest impact on performance improvement. Not only Lucene, but also the translog I/O adds up to the disk I/O with local gateway. The most interesting numbers are - the sustained total amount of data that can be indexed per time interval (MB per sec, not only the docs per sec) because this is determining the I/O throughput performance (sustained means, not just peak time for some seconds, but observed on average during a significant time where all nodes resources are fully utilized) - the load of the machine while the node is indexing (not CPU cycles) because this is significant for the overall workload throughput of the system - and the same while the node is being searched (or even both indexing and being searched) On big machines, like HPC servers, these numbers scale much better than on desktop machines. To get shard numbers that can be compared, performance should be measured against data volumes that utilize the machine to its fullest extent. That is, if RAM is around 16 G and heap is 8 G, indexing 4 G of data may not exploit all the machine's resources. So it's really a hard task to make statements like "ES can manage 20+ or 100+ shards per node". It always depends. There is always a compromise. Best regards, Jörg -- ---
**Author:** ![egaumer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/egaumer/32/2365_2.png) [@egaumer](https://discuss.elastic.co/u/egaumer)\ **Post date:** [January 21, 2013, 5:11pm UTC](https://discuss.elastic.co/t/understanding-threadpools/10417/7 "2013-01-21T17:11:13Z")
Yes, it's a factor of the hardware you're working with. In our case, we had SSDs with 10GigE and 128GB of RAM with 24 cores. Even with 20 shards (500GB total index size) the CPU load was under 50% and the I/O and network subsystems were not saturated. But in general, as Otis mentioned, more IndexWriters means more parallelism. If you're indexing throughput isn't going up and none of your resources are bound, then consider adding more shards. The concurrent flushing in Lucene 4 should reduce contention on the IndexWriter. On Monday, January 21, 2013 11:27:38 AM UTC-5, Jörg Prante wrote: > Increasing shard count per node scales well as long as > > - the IO subsystem is not satisfied > > - the CPU can manage the load created by the increasing number of threads > > The slowest part of the machine is the disk subsystem (if it's SSD or > spindle disk makes no big difference in that point). It means tuning the > disk I/O will give the highest impact on performance improvement. > > Not only Lucene, but also the translog I/O adds up to the disk I/O with > local gateway. > > The most interesting numbers are > > - the sustained total amount of data that can be indexed per time interval > (MB per sec, not only the docs per sec) because this is determining the I/O > throughput performance (sustained means, not just peak time for some > seconds, but observed on average during a significant time where all nodes > resources are fully utilized) > > - the load of the machine while the node is indexing (not CPU cycles) > because this is significant for the overall workload throughput of the > system > > - and the same while the node is being searched (or even both indexing and > being searched) > > On big machines, like HPC servers, these numbers scale much better than on > desktop machines. > > To get shard numbers that can be compared, performance should be measured > against data volumes that utilize the machine to its fullest extent. That > is, if RAM is around 16 G and heap is 8 G, indexing 4 G of data may not > exploit all the machine's resources. > > So it's really a hard task to make statements like "ES can manage 20+ or > 100+ shards per node". It always depends. There is always a compromise. > > Best regards, > > Jörg -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:55am UTC](https://discuss.elastic.co/t/understanding-threadpools/10417/8 "2017-07-06T02:55:23Z")