Elasticsearch high load/CPU usage

Crwe · September 30, 2012, 8:46pm

Thanks Shay. The log (and bigdesk) suggested ES was using 3.8GB out of
HEAP_SIZE=6GB, with field cache total of 2.9GB (I guess that's out of
the 3.8GB?). I thought 2GB free would be a healthy margin; any rules
of thumb on how much "free" RAM to leave inside the JVM?

Btw the machine has 12GB RAM total, so 6GB for java falls within your
~1/2 recommendation.

Cheers.

On Sep 30, 7:07 pm, Shay Banon kim...@gmail.com wrote:

Hey, please gist those responses, otherwise its hard to analyze them.

It seems like you don't have enough memory allocated to ES to do the type of faceting/sorting you are trying to do. Either allocate more memory to each ES instance (but make sure not to allocate more than ~1/2 of the machine memory, as a rule of thumb), or, increase the number of nodes in the cluster.

On Sep 30, 2012, at 7:16 AM, Crwe tester.teste...@gmail.com wrote:

I upgraded to ES 19.9. The CPU problem persists, but hot_threads
outputs more info now:

25.9% (129.5ms out of 500ms) cpu usage by thread
'elasticsearch[Rage][search][T#12]'
10/10 snapshots sharing following 22 elements

org.elasticsearch.index.field.data.strings.StringFieldData.load(StringField Data.java:
90)

org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringF ieldDataType.java:
56)

org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringF ieldDataType.java:
34)

org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)

org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapField DataCache.cache(AbstractConcurrentMapFieldDataCache.java:
129)

org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollec tor.doSetNextReader(TermsStringOrdinalsFacetCollector.java:
128)

org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(Abstrac tFacetCollector.java:
81)

org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector .java:
67)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)

org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexS earcher.java:
195)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)

org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:178)

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java :
286)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQue ry(SearchServiceTransportAction.java:
174)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction
$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:
148)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction
$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:135)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
java.lang.Thread.run(Unknown Source)

23.0% (114.9ms out of 500ms) cpu usage by thread
'elasticsearch[Rage][search][T#13]'
8/10 snapshots sharing following 23 elements

org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLo ader.java:
68)

org.elasticsearch.index.field.data.strings.StringFieldData.load(StringField Data.java:
90)

org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringF ieldDataType.java:
56)

org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringF ieldDataType.java:
34)

org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)

org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapField DataCache.cache(AbstractConcurrentMapFieldDataCache.java:
129)

org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollec tor.doSetNextReader(TermsStringOrdinalsFacetCollector.java:
128)

org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(Abstrac tFacetCollector.java:
81)

org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector .java:
67)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)

org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexS earcher.java:
195)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)

org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:178)

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java :
286)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQue ry(SearchServiceTransportAction.java:
174)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction
$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:
148)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction
$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:135)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
java.lang.Thread.run(Unknown Source)
2/10 snapshots sharing following 22 elements

org.elasticsearch.index.field.data.strings.StringFieldData.load(StringField Data.java:
90)

org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringF ieldDataType.java:
56)

org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringF ieldDataType.java:
34)

org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)

org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapField DataCache.cache(AbstractConcurrentMapFieldDataCache.java:
129)

org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollec tor.doSetNextReader(TermsStringOrdinalsFacetCollector.java:
128)

org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(Abstrac tFacetCollector.java:
81)

org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector .java:
67)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)

org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexS earcher.java:
195)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)

org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:178)

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java :
286)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQue ry(SearchServiceTransportAction.java:
174)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction
$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:
148)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction
$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:135)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
java.lang.Thread.run(Unknown Source)

21.4% (106.9ms out of 500ms) cpu usage by thread
'elasticsearch[Rage][search][T#11]'
5/10 snapshots sharing following 23 elements

org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLo ader.java:
79)

org.elasticsearch.index.field.data.strings.StringFieldData.load(StringField Data.java:
90)

org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringF ieldDataType.java:
56)

org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringF ieldDataType.java:
34)

org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)

org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapField DataCache.cache(AbstractConcurrentMapFieldDataCache.java:
129)

org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollec tor.doSetNextReader(TermsStringOrdinalsFacetCollector.java:
128)

org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(Abstrac tFacetCollector.java:
81)

org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector .java:
67)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)

org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexS earcher.java:
195)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)

org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)

org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:178)

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java :
286)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQue ry(SearchServiceTransportAction.java:
174)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction
$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:
148)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction
$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:135)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)

...

read more »

--

Topic		Replies	Views
Elasticsearch full CPU utillization Elasticsearch	2	841	July 6, 2017
Periodic CPU spikes Elasticsearch	9	2918	July 6, 2017
Investigating elasticsearch load issues Elasticsearch	9	1295	July 6, 2017
Help me debug CPU use issues Elasticsearch	15	1127	July 6, 2017
Optimize elasticsearch / JVM Elasticsearch	6	1339	July 6, 2017

Elasticsearch high load/CPU usage

Related topics