# 10% CPU load without querying **URL:** \ **Category:** Elasticsearch\ **Created:** [June 9, 2011, 11:26am UTC](https://discuss.elastic.co/t/10-cpu-load-without-querying/4587 "2011-06-09T11:26:09Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![maho](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@maho](https://discuss.elastic.co/u/maho)\ **Post date:** [June 9, 2011, 11:26am UTC](https://discuss.elastic.co/t/10-cpu-load-without-querying/4587/1 "2011-06-09T11:26:09Z")
Hi, i have 1 node with 1000 indices. Each index has 100k documents and 15 fields. If I start elasticsearch I have a very high cpu load of 100-200% (4x2,7ghz) for 5 minutes. After startup the cpu load is about 5-10%. Is that ok? ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 9, 2011, 12:06pm UTC](https://discuss.elastic.co/t/10-cpu-load-without-querying/4587/2 "2011-06-09T12:06:19Z")
Hi Maho On Thu, 2011-06-09 at 04:26 -0700, maho wrote: > Hi, > > i have 1 node with 1000 indices. Each index has 100k documents and 15 > fields. > If I start elasticsearch I have a very high cpu load of 100-200% > (4x2,7ghz) for 5 minutes. > > After startup the cpu load is about 5-10%. Is that ok? That sounds pretty normal, for a high number of indices. Given that each index is small, I'd set each on to have only one primary shard (number\_of\_shards) rather than the default 5. This will improve start-uptime, performance and memory usage. Also, why do you have so many indices? Could these not be combined? You may be able to make use of the new alias filter functionality that will be in 0.17: > > > An alias can now have a filter associated with it. Aliases with filters provide …an easy way to create different "views" of the same index. The filter can be defined using Query DSL and is applied to all Search, Count, Delete By Query and More Like This operations with this alias. > > \ > $ curl -XPOST 'http://localhost:9200/\_aliases' -d ' > { > "actions" : \[ > { > "add" : { > "index" : "test1", > "alias" : "alias2", > "filter" : { "term" : { "user" : "kimchy" } } > } > } > \] > }' > \ clint ---
**Author:** ![maho](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@maho](https://discuss.elastic.co/u/maho)\ **Post date:** [June 9, 2011, 2:23pm UTC](https://discuss.elastic.co/t/10-cpu-load-without-querying/4587/3 "2011-06-09T14:23:27Z")
Thanks for your answer. I've already set the shards per index to 1. Currently I'm evaluating solr and elasticsearch. In Solr, the startup with 1000 indices takes 1 or 2 minutes and after startup the cpu load is 0%. So I'm a little bit confused why elasticsearch is so resource intensive. Why 100 indices? Because every customer has its own index. Using alias filter could be a solution. But I think that would affect the scoring of the documents because the ranking algorithm will be calculated on the basis of all documents (=\> inverse document frequency)?! On 9 Jun., 14:06, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote: > Hi Maho > > On Thu, 2011-06-09 at 04:26 -0700, maho wrote: > > > Hi, > > > i have 1 node with 1000 indices. Each index has 100k documents and 15 > > fields. > > If I start elasticsearch I have a very high cpu load of 100-200% > > (4x2,7ghz) for 5 minutes. > > > After startup the cpu load is about 5-10%. Is that ok? > > That sounds pretty normal, for a high number of indices. > > Given that each index is small, I'd set each on to have only one primary > shard (number\_of\_shards) rather than the default 5. > > This will improve start-uptime, performance and memory usage. > > Also, why do you have so many indices? Could these not be combined? > > You may be able to make use of the new alias filter functionality that > will be in 0.17: > > [Aliases: add an ability to specify filters on aliases · Issue #971 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/971) > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 9, 2011, 11:46pm UTC](https://discuss.elastic.co/t/10-cpu-load-without-querying/4587/4 "2011-06-09T23:46:31Z")
After things have settles (cluster health is green), then you should not get high CPU load. I just did a quick test on my machine, and I get 0.2% usage with 1000 indices. The reason why recovery takes time and its resource intensive is mainly due to the fact that for each shard, the transaction log is replayed (elasticsearch does not require commit to be issued for data to be "safe"). You can issue a flush to flush the transaction log before you shutdown, and then there won't be anything to replay. On Thursday, June 9, 2011 at 5:23 PM, maho wrote: > Thanks for your answer. > > I've already set the shards per index to 1. > Currently I'm evaluating solr and elasticsearch. > In Solr, the startup with 1000 indices takes 1 or 2 minutes and after > startup the cpu load is 0%. > So I'm a little bit confused why elasticsearch is so resource > intensive. > > Why 100 indices? Because every customer has its own index. Using alias > filter could be a solution. But I think that would affect the scoring > of the documents because the ranking algorithm will be calculated on > the basis of all documents (=\> inverse document frequency)?! > > On 9 Jun., 14:06, Clinton Gormley \<[clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) ([http://iannounce.co.uk](http://iannounce.co.uk))\> wrote: > > > Hi Maho > > > > On Thu, 2011-06-09 at 04:26 -0700, maho wrote: > > > > > Hi, > > > > > i have 1 node with 1000 indices. Each index has 100k documents and 15 > > > fields. > > > If I start elasticsearch I have a very high cpu load of 100-200% > > > (4x2,7ghz) for 5 minutes. > > > > > After startup the cpu load is about 5-10%. Is that ok? > > > > That sounds pretty normal, for a high number of indices. > > > > Given that each index is small, I'd set each on to have only one primary > > shard (number\_of\_shards) rather than the default 5. > > > > This will improve start-uptime, performance and memory usage. > > > > Also, why do you have so many indices? Could these not be combined? > > > > You may be able to make use of the new alias filter functionality that > > will be in 0.17: > > > > [Aliases: add an ability to specify filters on aliases · Issue #971 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/971) > > > > clint ---
**Author:** ![maho](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@maho](https://discuss.elastic.co/u/maho)\ **Post date:** [June 10, 2011, 12:04pm UTC](https://discuss.elastic.co/t/10-cpu-load-without-querying/4587/5 "2011-06-10T12:04:47Z")
Now I deleted every index and created 1000 empty indices: 1000 opened indices: 8% CPU load (8 cpu cores, so it will be about 1,00%) 1000 closed indices: 0-1% CPU load (8 cpu cores, so it will be about 0,06%) The bad thing is, that the queries per second (JMeter test) decreased by 20% Is that ok or is something wrong with my installation / configuration / system? On 10 Jun., 01:46, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > After things have settles (cluster health is green), then you should not get high CPU load. I just did a quick test on my machine, and I get 0.2% usage with 1000 indices. > > The reason why recovery takes time and its resource intensive is mainly due to the fact that for each shard, the transaction log is replayed (elasticsearch does not require commit to be issued for data to be "safe"). You can issue a flush to flush the transaction log before you shutdown, and then there won't be anything to replay. > > On Thursday, June 9, 2011 at 5:23 PM, maho wrote: > > > Thanks for your answer. > > > I've already set the shards per index to 1. > > Currently I'm evaluating solr and elasticsearch. > > In Solr, the startup with 1000 indices takes 1 or 2 minutes and after > > startup the cpu load is 0%. > > So I'm a little bit confused why elasticsearch is so resource > > intensive. > > > Why 100 indices? Because every customer has its own index. Using alias > > filter could be a solution. But I think that would affect the scoring > > of the documents because the ranking algorithm will be calculated on > > the basis of all documents (=\> inverse document frequency)?! > > > On 9 Jun., 14:06, Clinton Gormley \<[clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) ([http://iannounce.co.uk](http://iannounce.co.uk))\> wrote: > > > > > Hi Maho > > > > On Thu, 2011-06-09 at 04:26 -0700, maho wrote: > > > > > > > Hi, > > > > > i have 1 node with 1000 indices. Each index has 100k documents and 15 > > > > fields. > > > > If I start elasticsearch I have a very high cpu load of 100-200% > > > > (4x2,7ghz) for 5 minutes. > > > > > After startup the cpu load is about 5-10%. Is that ok? > > > > That sounds pretty normal, for a high number of indices. > > > > Given that each index is small, I'd set each on to have only one primary > > > shard (number\_of\_shards) rather than the default 5. > > > > This will improve start-uptime, performance and memory usage. > > > > Also, why do you have so many indices? Could these not be combined? > > > > You may be able to make use of the new alias filter functionality that > > > will be in 0.17: > > > > [Aliases: add an ability to specify filters on aliases · Issue #971 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/971) > > > > clint ---
**Author:** ![maho](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@maho](https://discuss.elastic.co/u/maho)\ **Post date:** [June 11, 2011, 5:21pm UTC](https://discuss.elastic.co/t/10-cpu-load-without-querying/4587/6 "2011-06-11T17:21:58Z")
I now installed ES on my windows machine and the problems didn't appear. And I didn't get the DEBUG messages like in linux (used same config on both system): ... [2011-06-11 19:15:06,007[DEBUG][gateway.local] [Buzz] [core0][0]: throttling allocation [[core0][0], node[null]], [P], s[UNASSIGNED]] to [[Buzz][bfMpEcLeS-qBG8tT\_QN\_sw][inet[/10.0.2.15:9300]]] on primary allocation [2011-06-11 19:15:06,784[DEBUG][gateway.local] [Buzz] [core1][0]: throttling allocation [[core1][0], node[null]], [P], s[UNASSIGNED]] to [[Buzz][bfMpEcLeS-qBG8tT\_QN\_sw][inet[/10.0.2.15:9300]]] on primary allocation ... On 10 Jun., 14:04, maho [mathias.hod...@gmail.com](mailto:mathias.hod...@gmail.com) wrote: > Now I deleted every index and created 1000 empty indices: > > 1000 opened indices: 8% CPU load (8 cpu cores, so it will be about > 1,00%) > 1000 closed indices: 0-1% CPU load (8 cpu cores, so it will be about > 0,06%) > > The bad thing is, that the queries per second (JMeter test) decreased > by 20% > > Is that ok or is something wrong with my installation / > configuration / system? > > On10Jun., 01:46, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > After things have settles (cluster health is green), then you should not get high CPU load. I just did a quick test on my machine, and I get 0.2% usage with 1000 indices. > > > The reason why recovery takes time and its resource intensive is mainly due to the fact that for each shard, the transaction log is replayed (elasticsearch does not require commit to be issued for data to be "safe"). You can issue a flush to flush the transaction log before you shutdown, and then there won't be anything to replay. > > > On Thursday, June 9, 2011 at 5:23 PM, maho wrote: > > > > > Thanks for your answer. > > > > I've already set the shards per index to 1. > > > Currently I'm evaluating solr and elasticsearch. > > > In Solr, the startup with 1000 indices takes 1 or 2 minutes and after > > > startup the cpu load is 0%. > > > So I'm a little bit confused why elasticsearch is so resource > > > intensive. > > > > Why 100 indices? Because every customer has its own index. Using alias > > > filter could be a solution. But I think that would affect the scoring > > > of the documents because the ranking algorithm will be calculated on > > > the basis of all documents (=\> inverse document frequency)?! > > > > On 9 Jun., 14:06, Clinton Gormley \<[clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) ([http://iannounce.co.uk](http://iannounce.co.uk))\> wrote: > > > > > > > Hi Maho > > > > > On Thu, 2011-06-09 at 04:26 -0700, maho wrote: > > > > > > > > > Hi, > > > > > > i have 1 node with 1000 indices. Each index has 100k documents and 15 > > > > > fields. > > > > > If I start elasticsearch I have a very high cpu load of 100-200% > > > > > (4x2,7ghz) for 5 minutes. > > > > > > After startup the cpu load is about 5-10%. Is that ok? > > > > > That sounds pretty normal, for a high number of indices. > > > > > Given that each index is small, I'd set each on to have only one primary > > > > shard (number\_of\_shards) rather than the default 5. > > > > > This will improve start-uptime, performance and memory usage. > > > > > Also, why do you have so many indices? Could these not be combined? > > > > > You may be able to make use of the new alias filter functionality that > > > > will be in 0.17: > > > > > [Aliases: add an ability to specify filters on aliases · Issue #971 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/971) > > > > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 12, 2011, 7:13am UTC](https://discuss.elastic.co/t/10-cpu-load-without-querying/4587/7 "2011-06-12T07:13:27Z")
There isn't a difference between windows and linux in this case. The messages you see are elasticsearch throttling the concurrent allocation of shards on the same node so it won't be overloaded (and become unusable). By default, it allows for 4 concurrent allocations per node. On Saturday, June 11, 2011 at 8:21 PM, maho wrote: > I now installed ES on my windows machine and the problems didn't > appear. > And I didn't get the DEBUG messages like in linux (used same config on > both system): > > ... > [2011-06-11 19:15:06,007[DEBUG][gateway.local] [Buzz] [core0][0]: > throttling allocation [[core0][0], node[null]], [P], s[UNASSIGNED]] to > [[Buzz][bfMpEcLeS-qBG8tT\_QN\_sw][inet[/10.0.2.15:9300]]] on primary > allocation > > [2011-06-11 19:15:06,784[DEBUG][gateway.local] [Buzz] [core1][0]: > throttling allocation [[core1][0], node[null]], [P], s[UNASSIGNED]] to > [[Buzz][bfMpEcLeS-qBG8tT\_QN\_sw][inet[/10.0.2.15:9300]]] on primary > allocation > ... > > On 10 Jun., 14:04, maho \<[mathias.hod...@gmail.com](mailto:mathias.hod...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote: > > > Now I deleted every index and created 1000 empty indices: > > > > 1000 opened indices: 8% CPU load (8 cpu cores, so it will be about > > 1,00%) > > 1000 closed indices: 0-1% CPU load (8 cpu cores, so it will be about > > 0,06%) > > > > The bad thing is, that the queries per second (JMeter test) decreased > > by 20% > > > > Is that ok or is something wrong with my installation / > > configuration / system? > > > > On10Jun., 01:46, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > > > After things have settles (cluster health is green), then you should not get high CPU load. I just did a quick test on my machine, and I get 0.2% usage with 1000 indices. > > > > > The reason why recovery takes time and its resource intensive is mainly due to the fact that for each shard, the transaction log is replayed (elasticsearch does not require commit to be issued for data to be "safe"). You can issue a flush to flush the transaction log before you shutdown, and then there won't be anything to replay. > > > > > On Thursday, June 9, 2011 at 5:23 PM, maho wrote: > > > > > > > Thanks for your answer. > > > > > > I've already set the shards per index to 1. > > > > Currently I'm evaluating solr and elasticsearch. > > > > In Solr, the startup with 1000 indices takes 1 or 2 minutes and after > > > > startup the cpu load is 0%. > > > > So I'm a little bit confused why elasticsearch is so resource > > > > intensive. > > > > > > Why 100 indices? Because every customer has its own index. Using alias > > > > filter could be a solution. But I think that would affect the scoring > > > > of the documents because the ranking algorithm will be calculated on > > > > the basis of all documents (=\> inverse document frequency)?! > > > > > > On 9 Jun., 14:06, Clinton Gormley \<[clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) ([http://iannounce.co.uk](http://iannounce.co.uk)) ([http://iannounce.co.uk](http://iannounce.co.uk))\> wrote: > > > > > > > > > Hi Maho > > > > > > > On Thu, 2011-06-09 at 04:26 -0700, maho wrote: > > > > > > > > > > > Hi, > > > > > > > > i have 1 node with 1000 indices. Each index has 100k documents and 15 > > > > > > fields. > > > > > > If I start elasticsearch I have a very high cpu load of 100-200% > > > > > > (4x2,7ghz) for 5 minutes. > > > > > > > > After startup the cpu load is about 5-10%. Is that ok? > > > > > > > That sounds pretty normal, for a high number of indices. > > > > > > > Given that each index is small, I'd set each on to have only one primary > > > > > shard (number\_of\_shards) rather than the default 5. > > > > > > > This will improve start-uptime, performance and memory usage. > > > > > > > Also, why do you have so many indices? Could these not be combined? > > > > > > > You may be able to make use of the new alias filter functionality that > > > > > will be in 0.17: > > > > > > > [Aliases: add an ability to specify filters on aliases · Issue #971 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/971) > > > > > > > clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:03am UTC](https://discuss.elastic.co/t/10-cpu-load-without-querying/4587/8 "2017-07-06T04:03:47Z")