# High cpu load on load test with 300 rps **URL:** \ **Category:** Elasticsearch\ **Created:** [December 5, 2012, 6:07am UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945 "2012-12-05T06:07:36Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\ **Post date:** [December 5, 2012, 6:07am UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945/1 "2012-12-05T06:07:36Z")
hi,all my es configuration: 4 shard 1 replica 1 node 17646067 documents 8g index size es version: 0.19.11 java version: Java(TM) SE Runtime Environment (build 1.6.0\_37-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) os: CPU vendor: Intel CPU model: Xeon (2393 MHz) CPU total logical cores: 16 CPU cache: 12kb Total mem: 30.9gb (33271316480 b) Total swap: 3.9gb (4293586944 b) pressure: 300 search request per second One of my query: [2012-12-05 13:12:15,465][TRACE][index.search.slowlog.query] [Berzerker] [ptc][1] took[500.8ms], took\_millis[500], search\_type[QUERY\_THEN\_FETCH], total\_shards[4], source[{"from":0,"size":10,"timeout":5000,"query": {"query\_string":{"query":"出租车叫车","fields":["address^1.0","category.name^1.0","name^10.0","trade.name^1.0"],"default\_operator":"and","allow\_leading\_wildcard":false}},"filter":{"bool":{"must":{"term":{"location.cityId":"411200"}}}},"explain":false,"fields":["id","address","name"]}], extra\_source[] hprof cpu samples: see attachement hprof.cpu.samples.txt Yourkit profile snapshot: see attachment es3.png, the original snapshot is too large for attachement. It's not the first time to come across this problem with es and it's a very big problem for production usage. -- ---
**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\ **Post date:** [December 5, 2012, 6:28am UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945/2 "2012-12-05T06:28:28Z")
Additional info about es configuration: bootstrap.mlockall: true index: store: type: mmapfs analysis: analyzer: edgeNGramAnalyzer: type: custome tokenizer: standard filter: [standard,lowercase,englishSnowball,edgeNGramFilter] nGramAnalyzer: type: custome tokenizer: standard filter: [standard,lowercase,englishSnowball,nGramFilter] standardAnalyzer: type: custome tokenizer: standard filter: [standard,lowercase,englishSnowball] mmsegAnalyzer: type: custome tokenizer: mmseg\_maxword filter: [standard,lowercase,englishSnowball] complexAnalyzer: type: custome tokenizer: mmseg\_complex filter: [standard,lowercase,englishSnowball] simpleAnalyzer: type: custome tokenizer: mmseg\_simple filter: [standard,lowercase,englishSnowball] tokenizer: mmseg\_maxword: type: mmseg seg\_type: "max\_word" mmseg\_complex: type: mmseg seg\_type: "complex" mmseg\_simple: type: mmseg seg\_type: "simple" filter: nGramFilter: type: nGram min\_gram: 1 max\_gram: 64 edgeNGramFilter: type: edgeNGram min\_gram: 1 max\_gram: 64 side: front englishSnowball: type: snowball language: English here mmseg is a chinese analyzer, the homepage can be found here: [https://code.google.com/p/mmseg4j/](https://code.google.com/p/mmseg4j/) On Wednesday, December 5, 2012 2:07:36 PM UTC+8, Weiwei Wang wrote: > hi,all > my es configuration: > 4 shard > 1 replica > 1 node > 17646067 documents > 8g index size > > es version: > 0.19.11 > > java version: > Java(TM) SE Runtime Environment (build 1.6.0\_37-b06) > Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) > > os: > CPU vendor: Intel > CPU model: Xeon (2393 MHz) > CPU total logical cores: 16 > CPU cache: 12kb > Total mem: 30.9gb (33271316480 b) > Total swap: 3.9gb (4293586944 b) > > pressure: > 300 search request per second > > One of my query: > [2012-12-05 13:12:15,465][TRACE][index.search.slowlog.query] > [Berzerker] [ptc][1] took[500.8ms], took\_millis[500], > search\_type[QUERY\_THEN\_FETCH], total\_shards[4], > source[{"from":0,"size":10,"timeout":5000,"query": > {"query\_string":{"query":"出租车叫车","fields":["address^1.0","category.name > ^1.0","name^10.0","trade.name^1.0"],"default\_operator":"and","allow\_leading\_wildcard":false}},"filter":{"bool":{"must":{"term":{"location.cityId":"411200"}}}},"explain":false,"fields":["id","address","name"]}], > extra\_source > > hprof cpu samples: > see attachement hprof.cpu.samples.txt > > Yourkit profile snapshot: > see attachment es3.png, the original snapshot is too large for > attachement. > > It's not the first time to come across this problem with es and it's a > very big problem for production usage. -- ---
**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\ **Post date:** [December 5, 2012, 7:46am UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945/3 "2012-12-05T07:46:38Z")
Additional info about es configuration: bootstrap.mlockall: true index: store: type: mmapfs analysis: analyzer: edgeNGramAnalyzer: type: custome tokenizer: standard filter: [standard,lowercase,englishSnowball,edgeNGramFilter] nGramAnalyzer: type: custome tokenizer: standard filter: [standard,lowercase,englishSnowball,nGramFilter] standardAnalyzer: type: custome tokenizer: standard filter: [standard,lowercase,englishSnowball] mmsegAnalyzer: type: custome tokenizer: mmseg\_maxword filter: [standard,lowercase,englishSnowball] complexAnalyzer: type: custome tokenizer: mmseg\_complex filter: [standard,lowercase,englishSnowball] simpleAnalyzer: type: custome tokenizer: mmseg\_simple filter: [standard,lowercase,englishSnowball] tokenizer: mmseg\_maxword: type: mmseg seg\_type: "max\_word" mmseg\_complex: type: mmseg seg\_type: "complex" mmseg\_simple: type: mmseg seg\_type: "simple" filter: nGramFilter: type: nGram min\_gram: 1 max\_gram: 64 edgeNGramFilter: type: edgeNGram min\_gram: 1 max\_gram: 64 side: front englishSnowball: type: snowball language: English here mmseg is a chinese analyzer, the homepage can be found here: [https://code.google.com/p/mmseg4j/](https://code.google.com/p/mmseg4j/) cpu load is 1200% for 300rps. On Wednesday, December 5, 2012 2:07:36 PM UTC+8, Weiwei Wang wrote: > hi,all > my es configuration: > 4 shard > 1 replica > 1 node > 17646067 documents > 8g index size > > es version: > 0.19.11 > > java version: > Java(TM) SE Runtime Environment (build 1.6.0\_37-b06) > Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) > > os: > CPU vendor: Intel > CPU model: Xeon (2393 MHz) > CPU total logical cores: 16 > CPU cache: 12kb > Total mem: 30.9gb (33271316480 b) > Total swap: 3.9gb (4293586944 b) > > pressure: > 300 search request per second > > One of my query: > [2012-12-05 13:12:15,465][TRACE][index.search.slowlog.query] > [Berzerker] [ptc][1] took[500.8ms], took\_millis[500], > search\_type[QUERY\_THEN\_FETCH], total\_shards[4], > source[{"from":0,"size":10,"timeout":5000,"query": > {"query\_string":{"query":"出租车叫车","fields":["address^1.0","category.name > ^1.0","name^10.0","trade.name^1.0"],"default\_operator":"and","allow\_leading\_wildcard":false}},"filter":{"bool":{"must":{"term":{"location.cityId":"411200"}}}},"explain":false,"fields":["id","address","name"]}], > extra\_source > > hprof cpu samples: > see attachement hprof.cpu.samples.txt > > Yourkit profile snapshot: > see attachment es3.png, the original snapshot is too large for > attachement. > > It's not the first time to come across this problem with es and it's a > very big problem for production usage. -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 5, 2012, 10:55am UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945/4 "2012-12-05T10:55:41Z")
Hello, Just two questions and one remark: - how much memory do you allocate to ES? - what OS do you use? - I see you have min\_ngram=1 and max\_ngram=64. I would assume that would create lots of terms, and make your queries slow. Depending on how critical those settings are for the functionality of your application, I would look at shrinking theinterval, especially on the min\_ngram side ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Wed, Dec 5, 2012 at 9:46 AM, Weiwei Wang [ww.wang.cs@gmail.com](mailto:ww.wang.cs@gmail.com) wrote: > Additional info about es configuration: > > bootstrap.mlockall: true > index: > store: > type: mmapfs > analysis: > analyzer: > edgeNGramAnalyzer: > type: custome > tokenizer: standard > filter: [standard,lowercase, **englishSnowball,** > edgeNGramFilter] > nGramAnalyzer: > type: custome > tokenizer: standard > filter: [standard,lowercase,\*\*englishSnowball,nGramFilter] > standardAnalyzer: > type: custome > tokenizer: standard > filter: [standard,lowercase,\*\*englishSnowball] > mmsegAnalyzer: > type: custome > tokenizer: mmseg\_maxword > filter: [standard,lowercase,\*\*englishSnowball] > complexAnalyzer: > type: custome > tokenizer: mmseg\_complex > filter: [standard,lowercase,\*\*englishSnowball] > simpleAnalyzer: > type: custome > tokenizer: mmseg\_simple > filter: [standard,lowercase,\*\*englishSnowball] > tokenizer: > mmseg\_maxword: > type: mmseg > seg\_type: "max\_word" > mmseg\_complex: > type: mmseg > seg\_type: "complex" > mmseg\_simple: > type: mmseg > seg\_type: "simple" > filter: > nGramFilter: > type: nGram > min\_gram: 1 > max\_gram: 64 > edgeNGramFilter: > type: edgeNGram > min\_gram: 1 > max\_gram: 64 > side: front > englishSnowball: > type: snowball > language: English > > here mmseg is a chinese analyzer, the homepage can be found here: > [https://code.google.com/\*\*p/mmseg4j/](https://code.google.com/**p/mmseg4j/) [https://code.google.com/p/mmseg4j/](https://code.google.com/p/mmseg4j/) > > cpu load is 1200% for 300rps. > > On Wednesday, December 5, 2012 2:07:36 PM UTC+8, Weiwei Wang wrote: > > > hi,all > > my es configuration: > > 4 shard > > 1 replica > > 1 node > > 17646067 documents > > 8g index size > > > > es version: > > 0.19.11 > > > > java version: > > Java(TM) SE Runtime Environment (build 1.6.0\_37-b06) > > Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) > > > > os: > > CPU vendor: Intel > > CPU model: Xeon (2393 MHz) > > CPU total logical cores: 16 > > CPU cache: 12kb > > Total mem: 30.9gb (33271316480 b) > > Total swap: 3.9gb (4293586944 b) > > > > pressure: > > 300 search request per second > > > > One of my query: > > [2012-12-05 13:12:15,465][TRACE][index.**search.slowlog.query] > > [Berzerker] [ptc][1] took[500.8ms], took\_millis[500], > > search\_type[QUERY\_THEN\_FETCH], total\_shards[4], source[{"from":0,"size":10," > > timeout":5000,"query": {"query\_string":{"query":" > > 出租车叫车","fields":["address^1.0" **,"category.name^1.0","name^10.** 0"," > > trade.name^1.0"],"default\_operator":"and","allow\_ > > leading\_wildcard":false}},"filter":{"bool":{"must":{" > > term":{"location.cityId":"411200"}}}},"explain":false," > > fields":["id","address","name"**]}], extra\_source > > > > hprof cpu samples: > > see attachement hprof.cpu.samples.txt > > > > Yourkit profile snapshot: > > see attachment es3.png, the original snapshot is too large for > > attachement. > > > > It's not the first time to come across this problem with es and it's a > > very big problem for production usage. -- ---
**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\ **Post date:** [December 10, 2012, 3:24am UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945/5 "2012-12-10T03:24:48Z")
Hi, Radu, I attach the bigdesk snapshot and, I only use mmsegAnalyzer in my mapping, nGram and edgeNGram is not used now. Any more information please let me know. On Wednesday, December 5, 2012 6:55:41 PM UTC+8, Radu Gheorghe wrote: > Hello, > > Just two questions and one remark: > > - how much memory do you allocate to ES? > - what OS do you use? > - I see you have min\_ngram=1 and max\_ngram=64. I would assume that would > create lots of terms, and make your queries slow. Depending on how critical > those settings are for the functionality of your application, I would look > at shrinking theinterval, especially on the min\_ngram side > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Wed, Dec 5, 2012 at 9:46 AM, Weiwei Wang \<[ww.wa...@gmail.com](mailto:ww.wa...@gmail.com)\ > > > wrote: > > > Additional info about es configuration: > > > > bootstrap.mlockall: true > > index: > > store: > > type: mmapfs > > analysis: > > analyzer: > > edgeNGramAnalyzer: > > type: custome > > tokenizer: standard > > filter: [standard,lowercase, **englishSnowball,** > > edgeNGramFilter] > > nGramAnalyzer: > > type: custome > > tokenizer: standard > > filter: [standard,lowercase,\*\*englishSnowball,nGramFilter] > > standardAnalyzer: > > type: custome > > tokenizer: standard > > filter: [standard,lowercase,\*\*englishSnowball] > > mmsegAnalyzer: > > type: custome > > tokenizer: mmseg\_maxword > > filter: [standard,lowercase,\*\*englishSnowball] > > complexAnalyzer: > > type: custome > > tokenizer: mmseg\_complex > > filter: [standard,lowercase,\*\*englishSnowball] > > simpleAnalyzer: > > type: custome > > tokenizer: mmseg\_simple > > filter: [standard,lowercase,\*\*englishSnowball] > > tokenizer: > > mmseg\_maxword: > > type: mmseg > > seg\_type: "max\_word" > > mmseg\_complex: > > type: mmseg > > seg\_type: "complex" > > mmseg\_simple: > > type: mmseg > > seg\_type: "simple" > > filter: > > nGramFilter: > > type: nGram > > min\_gram: 1 > > max\_gram: 64 > > edgeNGramFilter: > > type: edgeNGram > > min\_gram: 1 > > max\_gram: 64 > > side: front > > englishSnowball: > > type: snowball > > language: English > > > > here mmseg is a chinese analyzer, the homepage can be found here: > > [https://code.google.com/\*\*p/mmseg4j/](https://code.google.com/**p/mmseg4j/) [https://code.google.com/p/mmseg4j/](https://code.google.com/p/mmseg4j/) > > > > cpu load is 1200% for 300rps. > > > > On Wednesday, December 5, 2012 2:07:36 PM UTC+8, Weiwei Wang wrote: > > > > > hi,all > > > my es configuration: > > > 4 shard > > > 1 replica > > > 1 node > > > 17646067 documents > > > 8g index size > > > > > > es version: > > > 0.19.11 > > > > > > java version: > > > Java(TM) SE Runtime Environment (build 1.6.0\_37-b06) > > > Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) > > > > > > os: > > > CPU vendor: Intel > > > CPU model: Xeon (2393 MHz) > > > CPU total logical cores: 16 > > > CPU cache: 12kb > > > Total mem: 30.9gb (33271316480 b) > > > Total swap: 3.9gb (4293586944 b) > > > > > > pressure: > > > 300 search request per second > > > > > > One of my query: > > > [2012-12-05 13:12:15,465][TRACE][index.**search.slowlog.query] > > > [Berzerker] [ptc][1] took[500.8ms], took\_millis[500], > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[4], source[{"from":0,"size":10," > > > timeout":5000,"query": {"query\_string":{"query":" > > > 出租车叫车","fields":["address^1.0" **,"category.name^1.0","name^10.** 0"," > > > trade.name^1.0"],"default\_operator":"and","allow\_ > > > leading\_wildcard":false}},"filter":{"bool":{"must":{" > > > term":{"location.cityId":"411200"}}}},"explain":false," > > > fields":["id","address","name"**]}], extra\_source > > > > > > hprof cpu samples: > > > see attachement hprof.cpu.samples.txt > > > > > > Yourkit profile snapshot: > > > see attachment es3.png, the original snapshot is too large for > > > attachement. > > > > > > It's not the first time to come across this problem with es and it's a > > > very big problem for production usage. -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 10, 2012, 3:58pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945/6 "2012-12-10T15:58:06Z")
Hello, Except for the high load, do you get any weird behavior, like ES getting unresponsive? Looking at the screenshots, I see more than 1K search requests per second (that would be your 300rps sent to each of your 4 shards), with more than 500 fetches, while the transport goes to 80-90MB/s both ways. That's a lot of load in my book, or maybe I'm missing something. That said, you might make things better by reducing the number of shards. That would hurt your indexing speed, though. Also, adding nodes and replicas (together) should help raise the number of concurrent queries your cluster can hold. And if you don't have a lot of indexing, you might benefit from optimizing your indices in off-peak intervals: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Mon, Dec 10, 2012 at 5:24 AM, Weiwei Wang [ww.wang.cs@gmail.com](mailto:ww.wang.cs@gmail.com) wrote: > Hi, Radu, > I attach the bigdesk snapshot and, I only use mmsegAnalyzer in my > mapping, nGram and edgeNGram is not used now. > > Any more information please let me know. > > On Wednesday, December 5, 2012 6:55:41 PM UTC+8, Radu Gheorghe wrote: > > > Hello, > > > > Just two questions and one remark: > > > > - how much memory do you allocate to ES? > > - what OS do you use? > > - I see you have min\_ngram=1 and max\_ngram=64. I would assume that would > > create lots of terms, and make your queries slow. Depending on how critical > > those settings are for the functionality of your application, I would look > > at shrinking theinterval, especially on the min\_ngram side > > > > ## Best regards, Radu > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > On Wed, Dec 5, 2012 at 9:46 AM, Weiwei Wang [ww.wa...@gmail.com](mailto:ww.wa...@gmail.com) wrote: > > > > > Additional info about es configuration: > > > > > > bootstrap.mlockall: true > > > index: > > > store: > > > type: mmapfs > > > analysis: > > > analyzer: > > > edgeNGramAnalyzer: > > > type: custome > > > tokenizer: standard > > > filter: [standard,lowercase, **englishSno** wball,\*\* > > > edgeNGramFilter] > > > nGramAnalyzer: > > > type: custome > > > tokenizer: standard > > > filter: [standard,lowercase, **englishSno** > > > wball,nGramFilter] > > > standardAnalyzer: > > > type: custome > > > tokenizer: standard > > > filter: [standard,lowercase, **englishSno** wball] > > > mmsegAnalyzer: > > > type: custome > > > tokenizer: mmseg\_maxword > > > filter: [standard,lowercase, **englishSno** wball] > > > complexAnalyzer: > > > type: custome > > > tokenizer: mmseg\_complex > > > filter: [standard,lowercase, **englishSno** wball] > > > simpleAnalyzer: > > > type: custome > > > tokenizer: mmseg\_simple > > > filter: [standard,lowercase, **englishSno** wball] > > > tokenizer: > > > mmseg\_maxword: > > > type: mmseg > > > seg\_type: "max\_word" > > > mmseg\_complex: > > > type: mmseg > > > seg\_type: "complex" > > > mmseg\_simple: > > > type: mmseg > > > seg\_type: "simple" > > > filter: > > > nGramFilter: > > > type: nGram > > > min\_gram: 1 > > > max\_gram: 64 > > > edgeNGramFilter: > > > type: edgeNGram > > > min\_gram: 1 > > > max\_gram: 64 > > > side: front > > > englishSnowball: > > > type: snowball > > > language: English > > > > > > here mmseg is a chinese analyzer, the homepage can be found here: > > > [https://code.google.com/\*\*p\*\*/mmseg4j/](https://code.google.com/ **p** /mmseg4j/)[https://code.google.com/p/mmseg4j/](https://code.google.com/p/mmseg4j/) > > > > > > cpu load is 1200% for 300rps. > > > > > > On Wednesday, December 5, 2012 2:07:36 PM UTC+8, Weiwei Wang wrote: > > > > > > > hi,all > > > > my es configuration: > > > > 4 shard > > > > 1 replica > > > > 1 node > > > > 17646067 documents > > > > 8g index size > > > > > > > > es version: > > > > 0.19.11 > > > > > > > > java version: > > > > Java(TM) SE Runtime Environment (build 1.6.0\_37-b06) > > > > Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) > > > > > > > > os: > > > > CPU vendor: Intel > > > > CPU model: Xeon (2393 MHz) > > > > CPU total logical cores: 16 > > > > CPU cache: 12kb > > > > Total mem: 30.9gb (33271316480 b) > > > > Total swap: 3.9gb (4293586944 b) > > > > > > > > pressure: > > > > 300 search request per second > > > > > > > > One of my query: > > > > [2012-12-05 13:12:15,465][TRACE][index. **sea** rch.slowlog.query] > > > > [Berzerker] [ptc][1] took[500.8ms], took\_millis[500], > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[4], source[{"from":0,"size":10," > > > > **ti** meout":5000,"query": {"query\_string":{"query":" **出租车** > > > > 叫车","fields":["address^1.0" **,"c** ategory.name [http://category.name](http://category.name) > > > > ^1.0","name^10. **0",**"trade.name^1.0"],"default\_ **ope** > > > > rator":"and","allow\_\*\*leading\_\*\*wildcard":false}}," **filter":{"** > > > > bool":{"must":{"\*\*term":{"**location.cityId":"411200"}}}}, > > > > "explain":false,"fields":["id","address","name"**]}], > > > > extra\_source > > > > > > > > hprof cpu samples: > > > > see attachement hprof.cpu.samples.txt > > > > > > > > Yourkit profile snapshot: > > > > see attachment es3.png, the original snapshot is too large for > > > > attachement. > > > > > > > > It's not the first time to come across this problem with es and it's a > > > > very big problem for production usage. -- ---
**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\ **Post date:** [December 18, 2012, 2:25am UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945/7 "2012-12-18T02:25:55Z")
Thanks, Radu, I use jmeter to run my load test, I confirmed I only configured load 300 rps, but the load is not directly to es but to my java program which use a NodeBuilder to build a elasticsearch node(client=true). My java program use a SearchRequestBuilder to do my search and each search return 10 results(query on multi field without highlight). ``` As you suggested in your post, I already add a node(now 2 nodes) and ``` the response time decays to around 100ms, however the cpu usage is still high on each node(now decay to around 600% for each node). ``` To my understanding, the cpu is used by lucene to do computaion and ``` currently I have no idea to decrease the cpu load instead of adding nodes and replica. ``` I still have a question about what is the maximum index size for a ``` shard? I need this information to set the number of shard. Currently i have 8g index size total(2g for each shard), take into the development of my product, the index size may increase to 80g or 100g. see the attachment for the jmeter conf On Monday, December 10, 2012 11:58:06 PM UTC+8, Radu Gheorghe wrote: > Hello, > > Except for the high load, do you get any weird behavior, like ES getting > unresponsive? > > Looking at the screenshots, I see more than 1K search requests per second > (that would be your 300rps sent to each of your 4 shards), with more than > 500 fetches, while the transport goes to 80-90MB/s both ways. That's a lot > of load in my book, or maybe I'm missing something. > > That said, you might make things better by reducing the number of shards. > That would hurt your indexing speed, though. Also, adding nodes and > replicas (together) should help raise the number of concurrent queries your > cluster can hold. > > And if you don't have a lot of indexing, you might benefit from optimizing > your indices in off-peak intervals: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-optimize.html) > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Mon, Dec 10, 2012 at 5:24 AM, Weiwei Wang \<[ww.wa...@gmail.com](mailto:ww.wa...@gmail.com)\ > > > wrote: > > > Hi, Radu, > > I attach the bigdesk snapshot and, I only use mmsegAnalyzer in my > > mapping, nGram and edgeNGram is not used now. > > > > Any more information please let me know. > > > > On Wednesday, December 5, 2012 6:55:41 PM UTC+8, Radu Gheorghe wrote: > > > > > Hello, > > > > > > Just two questions and one remark: > > > > > > - how much memory do you allocate to ES? > > > - what OS do you use? > > > - I see you have min\_ngram=1 and max\_ngram=64. I would assume that would > > > create lots of terms, and make your queries slow. Depending on how critical > > > those settings are for the functionality of your application, I would look > > > at shrinking theinterval, especially on the min\_ngram side > > > > > > ## Best regards, Radu > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > > On Wed, Dec 5, 2012 at 9:46 AM, Weiwei Wang [ww.wa...@gmail.com](mailto:ww.wa...@gmail.com) wrote: > > > > > > > Additional info about es configuration: > > > > > > > > bootstrap.mlockall: true > > > > index: > > > > store: > > > > type: mmapfs > > > > analysis: > > > > analyzer: > > > > edgeNGramAnalyzer: > > > > type: custome > > > > tokenizer: standard > > > > filter: [standard,lowercase, **englishSno** wball,\*\* > > > > edgeNGramFilter] > > > > nGramAnalyzer: > > > > type: custome > > > > tokenizer: standard > > > > filter: [standard,lowercase, **englishSno** > > > > wball,nGramFilter] > > > > standardAnalyzer: > > > > type: custome > > > > tokenizer: standard > > > > filter: [standard,lowercase, **englishSno** wball] > > > > mmsegAnalyzer: > > > > type: custome > > > > tokenizer: mmseg\_maxword > > > > filter: [standard,lowercase, **englishSno** wball] > > > > complexAnalyzer: > > > > type: custome > > > > tokenizer: mmseg\_complex > > > > filter: [standard,lowercase, **englishSno** wball] > > > > simpleAnalyzer: > > > > type: custome > > > > tokenizer: mmseg\_simple > > > > filter: [standard,lowercase, **englishSno** wball] > > > > tokenizer: > > > > mmseg\_maxword: > > > > type: mmseg > > > > seg\_type: "max\_word" > > > > mmseg\_complex: > > > > type: mmseg > > > > seg\_type: "complex" > > > > mmseg\_simple: > > > > type: mmseg > > > > seg\_type: "simple" > > > > filter: > > > > nGramFilter: > > > > type: nGram > > > > min\_gram: 1 > > > > max\_gram: 64 > > > > edgeNGramFilter: > > > > type: edgeNGram > > > > min\_gram: 1 > > > > max\_gram: 64 > > > > side: front > > > > englishSnowball: > > > > type: snowball > > > > language: English > > > > > > > > here mmseg is a chinese analyzer, the homepage can be found here: > > > > [https://code.google.com/\*\*p\*\*/mmseg4j/](https://code.google.com/ **p** /mmseg4j/)[https://code.google.com/p/mmseg4j/](https://code.google.com/p/mmseg4j/) > > > > > > > > cpu load is 1200% for 300rps. > > > > > > > > On Wednesday, December 5, 2012 2:07:36 PM UTC+8, Weiwei Wang wrote: > > > > > > > > > hi,all > > > > > my es configuration: > > > > > 4 shard > > > > > 1 replica > > > > > 1 node > > > > > 17646067 documents > > > > > 8g index size > > > > > > > > > > es version: > > > > > 0.19.11 > > > > > > > > > > java version: > > > > > Java(TM) SE Runtime Environment (build 1.6.0\_37-b06) > > > > > Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) > > > > > > > > > > os: > > > > > CPU vendor: Intel > > > > > CPU model: Xeon (2393 MHz) > > > > > CPU total logical cores: 16 > > > > > CPU cache: 12kb > > > > > Total mem: 30.9gb (33271316480 b) > > > > > Total swap: 3.9gb (4293586944 b) > > > > > > > > > > pressure: > > > > > 300 search request per second > > > > > > > > > > One of my query: > > > > > [2012-12-05 13:12:15,465][TRACE][index. **sea** rch.slowlog.query] > > > > > [Berzerker] [ptc][1] took[500.8ms], took\_millis[500], > > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[4], source[{"from":0,"size":10," > > > > > **ti** meout":5000,"query": {"query\_string":{"query":" **出租车** > > > > > 叫车","fields":["address^1.0" **,"c** ategory.name [http://category.name](http://category.name) > > > > > ^1.0","name^10. **0",**"trade.name^1.0"],"default\_ **ope** > > > > > rator":"and","allow\_\*\*leading\_\*\*wildcard":false}}," **filter":{"** > > > > > bool":{"must":{"\*\*term":{"**location.cityId":"411200"}}}}, > > > > > "explain":false,"fields":["id","address","name"**]}], > > > > > extra\_source > > > > > > > > > > hprof cpu samples: > > > > > see attachement hprof.cpu.samples.txt > > > > > > > > > > Yourkit profile snapshot: > > > > > see attachment es3.png, the original snapshot is too large for > > > > > attachement. > > > > > > > > > > It's not the first time to come across this problem with es and it's a > > > > > very big problem for production usage. -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 18, 2012, 2:54pm UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945/8 "2012-12-18T14:54:31Z")
Hello Weiwei, Good to know that adding nodes and replicas has such a good impact on performance. At this point I don't have any good ideas on how to further improve performance. Maybe upgrading to Java 1.7.x would help? I think it's worth a test. As for maximum shard size, AFAIK it depends on: - how your data looks like (how many documents, how many terms) - heap size So again, you have to test to be sure. But a high-level look brings good news: now you have 8G and ES used ~1G of heap, and goes to ~2G at times. So if you configure it to allocate about half of your total RAM (which is usually recommended), it should work with a single, 80G shard. ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Tue, Dec 18, 2012 at 4:25 AM, Weiwei Wang [ww.wang.cs@gmail.com](mailto:ww.wang.cs@gmail.com) wrote: > Thanks, Radu, > I use jmeter to run my load test, I confirmed I only configured > load 300 rps, but the load is not directly to es but to my java program > which use a NodeBuilder to build a elasticsearch node(client=true). My java > program use a SearchRequestBuilder to do my search and each search return > 10 results(query on multi field without highlight). > > ``` > As you suggested in your post, I already add a node(now 2 nodes) and > > ``` > > the response time decays to around 100ms, however the cpu usage is still > high on each node(now decay to around 600% for each node). > > ``` > To my understanding, the cpu is used by lucene to do computaion and > > ``` > > currently I have no idea to decrease the cpu load instead of adding nodes > and replica. > > ``` > I still have a question about what is the maximum index size for a > > ``` > > shard? I need this information to set the number of shard. Currently i have > 8g index size total(2g for each shard), take into the development of my > product, the index size may increase to 80g or 100g. > > see the attachment for the jmeter conf > > On Monday, December 10, 2012 11:58:06 PM UTC+8, Radu Gheorghe wrote: > > > Hello, > > > > Except for the high load, do you get any weird behavior, like ES getting > > unresponsive? > > > > Looking at the screenshots, I see more than 1K search requests per second > > (that would be your 300rps sent to each of your 4 shards), with more than > > 500 fetches, while the transport goes to 80-90MB/s both ways. That's a lot > > of load in my book, or maybe I'm missing something. > > > > That said, you might make things better by reducing the number of shards. > > That would hurt your indexing speed, though. Also, adding nodes and > > replicas (together) should help raise the number of concurrent queries your > > cluster can hold. > > > > And if you don't have a lot of indexing, you might benefit from > > optimizing your indices in off-peak intervals: > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/**guide/reference/api/admin-)\*\* > > indices-optimize.html[http://www.elasticsearch.org/guide/reference/api/admin-indices-optimize.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-optimize.html) > > > > ## Best regards, Radu > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > On Mon, Dec 10, 2012 at 5:24 AM, Weiwei Wang [ww.wa...@gmail.com](mailto:ww.wa...@gmail.com) wrote: > > > > > Hi, Radu, > > > I attach the bigdesk snapshot and, I only use mmsegAnalyzer in my > > > mapping, nGram and edgeNGram is not used now. > > > > > > Any more information please let me know. > > > > > > On Wednesday, December 5, 2012 6:55:41 PM UTC+8, Radu Gheorghe wrote: > > > > > > > Hello, > > > > > > > > Just two questions and one remark: > > > > > > > > - how much memory do you allocate to ES? > > > > - what OS do you use? > > > > - I see you have min\_ngram=1 and max\_ngram=64. I would assume that > > > > would create lots of terms, and make your queries slow. Depending on how > > > > critical those settings are for the functionality of your application, I > > > > would look at shrinking theinterval, especially on the min\_ngram side > > > > > > > > ## Best regards, Radu > > > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > > > > On Wed, Dec 5, 2012 at 9:46 AM, Weiwei Wang [ww.wa...@gmail.com](mailto:ww.wa...@gmail.com) wrote: > > > > > > > > > Additional info about es configuration: > > > > > > > > > > bootstrap.mlockall: true > > > > > index: > > > > > store: > > > > > type: mmapfs > > > > > analysis: > > > > > analyzer: > > > > > edgeNGramAnalyzer: > > > > > type: custome > > > > > tokenizer: standard > > > > > filter: [standard,lowercase, **englishSno\*\*\*\*wball,** > > > > > edgeNGramFilter] > > > > > nGramAnalyzer: > > > > > type: custome > > > > > tokenizer: standard > > > > > filter: [standard,lowercase, **englishSno** \*\* > > > > > wball,nGramFilter] > > > > > standardAnalyzer: > > > > > type: custome > > > > > tokenizer: standard > > > > > filter: [standard,lowercase,\*\*englishSno**wball] > > > > > mmsegAnalyzer: > > > > > type: custome > > > > > tokenizer: mmseg\_maxword > > > > > filter: [standard,lowercase,\*\*englishSno**wball] > > > > > complexAnalyzer: > > > > > type: custome > > > > > tokenizer: mmseg\_complex > > > > > filter: [standard,lowercase,\*\*englishSno**wball] > > > > > simpleAnalyzer: > > > > > type: custome > > > > > tokenizer: mmseg\_simple > > > > > filter: [standard,lowercase,\*\*englishSno**wball] > > > > > tokenizer: > > > > > mmseg\_maxword: > > > > > type: mmseg > > > > > seg\_type: "max\_word" > > > > > mmseg\_complex: > > > > > type: mmseg > > > > > seg\_type: "complex" > > > > > mmseg\_simple: > > > > > type: mmseg > > > > > seg\_type: "simple" > > > > > filter: > > > > > nGramFilter: > > > > > type: nGram > > > > > min\_gram: 1 > > > > > max\_gram: 64 > > > > > edgeNGramFilter: > > > > > type: edgeNGram > > > > > min\_gram: 1 > > > > > max\_gram: 64 > > > > > side: front > > > > > englishSnowball: > > > > > type: snowball > > > > > language: English > > > > > > > > > > here mmseg is a chinese analyzer, the homepage can be found here: > > > > > [https://code.google.com/\*\*p\*\*\*\*/mmseg4j/](https://code.google.com/ **p**** /mmseg4j/)[https://code.google.com/p/mmseg4j/](https://code.google.com/p/mmseg4j/) > > > > > > > > > > cpu load is 1200% for 300rps. > > > > > > > > > > On Wednesday, December 5, 2012 2:07:36 PM UTC+8, Weiwei Wang wrote: > > > > > > > > > > > hi,all > > > > > > my es configuration: > > > > > > 4 shard > > > > > > 1 replica > > > > > > 1 node > > > > > > 17646067 documents > > > > > > 8g index size > > > > > > > > > > > > es version: > > > > > > 0.19.11 > > > > > > > > > > > > java version: > > > > > > Java(TM) SE Runtime Environment (build 1.6.0\_37-b06) > > > > > > Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) > > > > > > > > > > > > os: > > > > > > CPU vendor: Intel > > > > > > CPU model: Xeon (2393 MHz) > > > > > > CPU total logical cores: 16 > > > > > > CPU cache: 12kb > > > > > > Total mem: 30.9gb (33271316480 b) > > > > > > Total swap: 3.9gb (4293586944 b) > > > > > > > > > > > > pressure: > > > > > > 300 search request per second > > > > > > > > > > > > One of my query: > > > > > > [2012-12-05 13:12:15,465][TRACE][index.**search.slowlog.query] > > > > > > [Berzerker] [ptc][1] took[500.8ms], took\_millis[500], > > > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[4], source[{"from":0,"size":10," > > > > > > \*\*timeout":5000,"query": {"query\_string":{"query":"出租车** > > > > > > 叫车","fields":["address^1.0"\*\*,"c**ategory.name[http://category.name](http://category.name) > > > > > > ^1.0","name^10.\*\*0",**"trade.name^1.0"],"default\_**ope**\*\* > > > > > > rator":"and","allow\_**leading\_wildcard":false}},"filter":{" > > > > > > boo **l":{"must":{"\*\*term":{"\*\*location.** cityId":"411200"}}}}, > > > > > > "explain" **:false,"** fields":["id" **,"** address","name"**]}], > > > > > > extra\_source > > > > > > > > > > > > hprof cpu samples: > > > > > > see attachement hprof.cpu.samples.txt > > > > > > > > > > > > Yourkit profile snapshot: > > > > > > see attachment es3.png, the original snapshot is too large for > > > > > > attachement. > > > > > > > > > > > > It's not the first time to come across this problem with es and it's > > > > > > a very big problem for production usage. > > > > > > > > ``` > > > > -- > > > > > > > > ``` -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:59am UTC](https://discuss.elastic.co/t/high-cpu-load-on-load-test-with-300-rps/9945/9 "2017-07-06T02:59:28Z")