# How to reduce elasticsearch high load average/cpu usage?

**URL:** <https://discuss.elastic.co/t/how-to-reduce-elasticsearch-high-load-average-cpu-usage/19932>\
**Category:** Elasticsearch\
**Created:** [September 24, 2014, 6:13am UTC](https://discuss.elastic.co/t/how-to-reduce-elasticsearch-high-load-average-cpu-usage/19932 "2014-09-24T06:13:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lam](https://avatars.discourse-cdn.com/v4/letter/l/6bbea6/32.png) [@lam](https://discuss.elastic.co/u/lam)\
**Post date:** [September 24, 2014, 6:13am UTC](https://discuss.elastic.co/t/how-to-reduce-elasticsearch-high-load-average-cpu-usage/19932/1 "2014-09-24T06:13:17Z")

</div>

I am a developer from CHINA, i was suffering from load average  
recently,which range is 2-5 .I have 5 clustered nodes and each nodes have 1  
replica,the cluster total document size is 2G and 2,000,000 docs.

Here are some relevant information:

_0.JVM_

> ps -aux | grep java  
> Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ  
> root 1232 380 37.6 11648332 3031468 ? Sl 10:40 742:38 /usr/bin/java -Xms6g -Xmx6g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Delasticsearch -Des.path.home=/usr/local/es1x/elasticsearch-1.3.2 -cp :/usr/local/es1x/elasticsearch-1.3.2/lib/elasticsearch-1.3.2.jar:/usr/local/es1x/elasticsearch-1.3.2/lib/_:/usr/local/es1x/elasticsearch-1.3.2/lib/sigar/_ org.elasticsearch.bootstrap.Elasticsearch  
> root 3016 0.0 0.0 103244 864 pts/1 S+ 13:55 0:00 grep java

_1.load average_

> top - 15:34:14 up 9 days, 23:09, 1 user, load average: 2.18, 2.30, 2.39  
> Tasks: 169 total, 1 running, 168 sleeping, 0 stopped, 0 zombie  
> Cpu0 : 51.8%us, 0.7%sy, 0.0%ni, 47.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st  
> Cpu1 : 46.3%us, 0.3%sy, 0.0%ni, 53.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st  
> Cpu2 : 37.0%us, 0.3%sy, 0.0%ni, 62.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st  
> Cpu3 : 35.6%us, 0.3%sy, 0.0%ni, 63.7%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st  
> Cpu4 : 47.5%us, 0.7%sy, 0.0%ni, 51.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st  
> Cpu5 : 33.3%us, 0.0%sy, 0.0%ni, 66.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st  
> Cpu6 : 25.4%us, 0.3%sy, 0.0%ni, 74.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st  
> Cpu7 : 12.7%us, 0.3%sy, 0.0%ni, 87.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> 
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND  
> 16637 root 20 0 9297m 3.6g 31m S 209.4 62.8 8964:38 java  
> 1 root 20 0 19232 1012 840 S 0.0 0.0 0:38.37 init

\*2.bin/elasticsearch -v \*

> Version: 1.3.2, Build: dee175d/2014-08-13T14:29:30Z, JVM: 1.7.0\_55

_3.java -version_

> java version "1.7.0\_55"  
> Java(TM) SE Runtime Environment (build 1.7.0\_55-b13)  
> Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)

_4、elasticsearch.yml_

> cluster.name: XXXXXX  
> node.name: 035
> 
> index.cache.field.max\_size: 500000  
> index.cache.field.expire: 5m  
> index:  
> analysis:  
> analyzer:  
> index\_ansj:  
> alias: [ansj\_index\_analyzer]  
> type: ansj\_index  
> user\_path: ansj/user  
> ambiguity: ansj/ambiguity.dic  
> stop\_path: ansj/stopLibrary.dic  
> is\_name: false  
> redis:  
> pool:  
> maxactive: 20  
> maxidle: 10  
> maxwait: 100  
> testonborrow: true  
> ip: 192.168.0.159:6379  
> channel: ansj\_term  
> query\_ansj:  
> alias: [ansj\_index\_analyzer]  
> type: ansj\_query  
> user\_path: ansj/user  
> ambiguity: ansj/ambiguity.dic  
> stop\_path: ansj/stopLibrary.dic  
> is\_name: false  
> redis:  
> pool:  
> maxactive: 20  
> maxidle: 10  
> maxwait: 100  
> testonborrow: true  
> ip: 192.168.0.159:6379  
> channel: ansj\_term  
> index.analysis.analyzer.default.type: keyword  
> ################################## Slow Log ##################################
> 
> index.search.slowlog.threshold.query.warn: 10s  
> index.search.slowlog.threshold.query.info: 5s
> 
> # index.search.slowlog.threshold.query.debug: 2s
> 
> # index.search.slowlog.threshold.query.trace: 500ms
> 
> index.search.slowlog.threshold.fetch.warn: 1s  
> index.search.slowlog.threshold.fetch.info: 800ms  
> index.search.slowlog.threshold.fetch.debug: 500ms  
> index.search.slowlog.threshold.fetch.trace: 200ms
> 
> index.indexing.slowlog.threshold.index.warn: 10s  
> index.indexing.slowlog.threshold.index.info: 5s  
> index.indexing.slowlog.threshold.index.debug: 2s  
> index.indexing.slowlog.threshold.index.trace: 500ms
> 
> ################################## GC Logging ################################  
> monitor.jvm.gc.young.warn: 1000ms  
> monitor.jvm.gc.young.info: 700ms  
> monitor.jvm.gc.young.debug: 400ms
> 
> monitor.jvm.gc.old.warn: 10s  
> monitor.jvm.gc.old.info: 5s  
> monitor.jvm.gc.old.debug: 2s
> 
> threadpool:  
> index:  
> type: fixed  
> size: 30  
> queue\_size: -1  
> search:  
> type: fixed  
> size: 30  
> queue\_size: 1000

_5.curl -XGET 'localhost:9200/\_nodes/hot\_threads'_

> ::: [180][VNscyuhPS3u94QuyI2TfPQ][es180][inet[/192.168.0.180:9300]]
> 
> 96.9% (484.5ms out of 500ms) cpu usage by thread 'elasticsearch[180][search][T#23]'  
> 2/10 snapshots sharing following 29 elements  
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)  
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)  
> Script1.run(Script1.groovy:1)  
> org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:252)  
> org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.runAsDouble(GroovyScriptEngineService.java:273)  
> org.elasticsearch.common.lucene.search.function.ScriptScoreFunction.score(ScriptScoreFunction.java:54)  
> org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorScorer.score(FunctionScoreQuery.java:175)  
> org.apache.lucene.search.FilteredQuery$LeapFrogScorer.score(FilteredQuery.java:308)  
> org.apache.lucene.search.ScoreCachingWrappingScorer.score(ScoreCachingWrappingScorer.java:49)  
> org.apache.lucene.search.FieldComparator$RelevanceComparator.compareBottom(FieldComparator.java:774)  
> org.apache.lucene.search.TopFieldCollector$OutOfOrderMultiComparatorNonScoringCollector.collect(TopFieldCollector.java:484)  
> org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:61)  
> org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)  
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)  
> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)  
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)  
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149)  
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:261)  
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)  
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)  
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 4/10 snapshots sharing following 16 elements  
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)  
> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)  
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)  
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149)  
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:261)  
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)  
> org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)  
> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 4/10 snapshots sharing following 2 elements  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 92.9% (464.4ms out of 500ms) cpu usage by thread 'elasticsearch[180][search][T#18]'  
> 10/10 snapshots sharing following 10 elements  
> sun.misc.Unsafe.park(Native Method)  
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)  
> java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)  
> java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)  
> java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)  
> org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)  
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 67.6% (337.9ms out of 500ms) cpu usage by thread 'elasticsearch[180][search][T#8]'  
> 7/10 snapshots sharing following 28 elements  
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)  
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)  
> Script9.run(Script9.groovy:1)  
> org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:252)  
> org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.runAsDouble(GroovyScriptEngineService.java:273)  
> org.elasticsearch.common.lucene.search.function.ScriptScoreFunction.score(ScriptScoreFunction.java:54)  
> org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorScorer.score(FunctionScoreQuery.java:175)  
> org.apache.lucene.search.ScoreCachingWrappingScorer.score(ScoreCachingWrappingScorer.java:49)  
> org.apache.lucene.search.FieldComparator$RelevanceComparator.compareBottom(FieldComparator.java:774)  
> org.apache.lucene.search.TopFieldCollector$OutOfOrderMultiComparatorNonScoringCollector.collect(TopFieldCollector.java:484)  
> org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:61)  
> org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)  
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)  
> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)  
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)  
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149)  
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:261)  
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:688)  
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:677)  
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)  
> 3/10 snapshots sharing following 16 elements  
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)  
> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)  
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)  
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)  
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149)  
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:261)  
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:688)  
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:677)  
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> java.lang.Thread.run(Thread.java:745)

Thank you!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/af4cc9c5-5939-4c0a-ad13-4383ba2978fd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/af4cc9c5-5939-4c0a-ad13-4383ba2978fd%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 24, 2014, 6:21am UTC](https://discuss.elastic.co/t/how-to-reduce-elasticsearch-high-load-average-cpu-usage/19932/2 "2014-09-24T06:21:42Z")

</div>

Sounds like you are using a plugin, right?  
Also you seem to run function score queries. How your queries look like?

Then, it's better to use GIST to attach long logs/files instead of pasting them here.

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 24 sept. 2014 à 08:13, 林 [lamhomemoon@gmail.com](mailto:lamhomemoon@gmail.com) a écrit :

I am a developer from CHINA, i was suffering from load average recently,which range is 2-5 .I have 5 clustered nodes and each nodes have 1 replica,the cluster total document size is 2G and 2,000,000 docs.

Here are some relevant information:

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/71B47A2E-51F0-4583-95BF-C8B004750FF9%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/71B47A2E-51F0-4583-95BF-C8B004750FF9%40pilato.fr).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![lam](https://avatars.discourse-cdn.com/v4/letter/l/6bbea6/32.png) [@lam](https://discuss.elastic.co/u/lam)\
**Post date:** [September 25, 2014, 4:00am UTC](https://discuss.elastic.co/t/how-to-reduce-elasticsearch-high-load-average-cpu-usage/19932/3 "2014-09-25T04:00:30Z")

</div>

Thanks for your advice.🙂

I use a quite complex query like that, which includes function score:

"script\_score": {  
"script":  
"((doc['updatetime'].value\>1411574400000?1411574400000:doc['updatetime'].value)/1000)+(\_score_43200\>43200?43200:\_score_43200)+(doc['column1'].value==1?14400:0)

- (doc['column2'].value==3?7200:0) +  
(doc['column2'].value==7?3600:0)+((doc['updatetime'].value\>1411574400000?1411574400000-doc['updatetime'].value:doc['updatetime'].value-1411574400000)/60000)",  
"lang": "groovy"  
},

The plugin used is an analyzer for chinese.

在 2014年9月24日星期三UTC+8下午2时21分56秒，David Pilato写道：

> Sounds like you are using a plugin, right?  
> Also you seem to run function score queries. How your queries look like?
> 
> Then, it's better to use GIST to attach long logs/files instead of pasting  
> them here.
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 24 sept. 2014 à 08:13, 林 \<[lamho...@gmail.com](mailto:lamho...@gmail.com) \<javascript:\>\> a écrit :
> 
> I am a developer from CHINA, i was suffering from load average  
> recently,which range is 2-5 .I have 5 clustered nodes and each nodes have 1  
> replica,the cluster total document size is 2G and 2,000,000 docs.
> 
> Here are some relevant information:

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/905e539a-ffa4-46a3-b348-a4b388871b66%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/905e539a-ffa4-46a3-b348-a4b388871b66%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:00am UTC](https://discuss.elastic.co/t/how-to-reduce-elasticsearch-high-load-average-cpu-usage/19932/4 "2017-07-06T01:00:07Z")

</div>