# Massive performance issues on our production cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [June 16, 2015, 11:12am UTC](https://discuss.elastic.co/t/massive-performance-issues-on-our-production-cluster/2785 "2015-06-16T11:12:18Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![salat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/salat/32/690_2.png) [@salat](https://discuss.elastic.co/u/salat)\ **Post date:** [June 16, 2015, 11:12am UTC](https://discuss.elastic.co/t/massive-performance-issues-on-our-production-cluster/2785/1 "2015-06-16T11:12:18Z")
Dear Folks, right now we see some serious performance issues on our AWS EC2 elasticsearch cluster. Our nodes are not responding very wenn and hanging at a **CPU load of in average 90%**. Here is a copy of the hot\_threads output: * * * _::: [ip-XXX-XXX-XXX-XXX][e9oLvVBiRUmK6oW2aIVrqQ][ip-XXX-XXX-XXX-XXX][inet[ip-XXX-XXX-XXX-XXX.eu-west-1.compute.internal/XXX-XXX-XXX-XXX:9300]]_ _Hot threads at 2015-06-16T11:00:47.638Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:_ **32.5% (162.3ms out of 500ms) cpu usage by thread 'elasticsearch[ip-XXX-XXX-XXX-XXX][search][T#1]'** _2/10 snapshots sharing following 18 elements_ _org.elasticsearch.search.aggregations.bucket.filter.FilterAggregator.collect(FilterAggregator.java:61)_ _org.elasticsearch.search.aggregations.AggregationPhase$AggregationsCollector.collect(AggregationPhase.java:161)_ _org.elasticsearch.common.lucene.MultiCollector.collect(MultiCollector.java:60)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)_ _org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)_ _org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)_ _org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:117)_ _org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)_ _org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)_ _org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)_ _org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)_ _org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ **7/10 snapshots sharing following 10 elements** _sun.misc.Unsafe.park(Native Method)_ _java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)_ _java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)_ _java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)_ _java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)_ _org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)_ _java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ **unique snapshot** _org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)_ _org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:101)_ _org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:135)_ _org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:354)_ _org.apache.lucene.index.SegmentReader.document(SegmentReader.java:335)_ _org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:407)_ _org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:217)_ _org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:182)_ _org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:501)_ _org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)_ _org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)_ _org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ _::: [ip-XXX-XXX-XXX-XXX][ZZfbwauFSZq8XXDnyVug4g][ip-XXX-XXX-XXX-XXX][inet[/XXX-XXX-XXX-XXX:9300]]_ _Hot threads at 2015-06-16T11:00:47.636Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:_ **33.7% (168.6ms out of 500ms) cpu usage by thread 'elasticsearch[ip-XXX-XXX-XXX-XXX][search][T#5]'** _2/10 snapshots sharing following 18 elements_ _org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(FilteredDocIdSetIterator.java:60)_ _org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:257)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)_ _org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)_ _org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)_ _org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:117)_ _org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)_ _org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)_ _org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)_ _org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ **4/10 snapshots sharing following 18 elements** _org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:248)_ _org.elasticsearch.search.aggregations.bucket.filter.FilterAggregator.setNextReader(FilterAggregator.java:52)_ _org.elasticsearch.search.aggregations.support.AggregationContext.setNextReader(AggregationContext.java:94)_ _org.elasticsearch.search.aggregations.AggregationPhase$AggregationsCollector.setNextReader(AggregationPhase.java:167)_ _org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)_ _org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)_ _org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:117)_ _org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)_ _org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)_ _org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)_ _org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ **4/10 snapshots sharing following 12 elements** _org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)_ _org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:117)_ _org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)_ _org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)_ _org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)_ _org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ _33.2% (165.7ms out of 500ms) cpu usage by thread 'elasticsearch[ip-XXX-XXX-XXX-XXX][search][T#1]'_ **3/10 snapshots sharing following 20 elements** _org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:257)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)_ _org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)_ _org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)_ _org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150)_ _org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)_ _org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)_ _org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)_ _org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ **7/10 snapshots sharing following 9 elements** _org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)_ _org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)_ _org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)_ _org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ _33.0% (165.1ms out of 500ms) cpu usage by thread 'elasticsearch[ip-XXX-XXX-XXX-XXX][search][T#2]'_ **4/10 snapshots sharing following 20 elements** _org.elasticsearch.search.aggregations.bucket.BucketsAggregator.collectBucket(BucketsAggregator.java:55)_ _org.elasticsearch.search.aggregations.bucket.filter.FilterAggregator.collect(FilterAggregator.java:61)_ _org.elasticsearch.search.aggregations.AggregationPhase$AggregationsCollector.collect(AggregationPhase.java:161)_ _org.elasticsearch.common.lucene.MultiCollector.collect(MultiCollector.java:60)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)_ _org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)_ _org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)_ _org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:117)_ _org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)_ _org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)_ _org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)_ _org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ **3/10 snapshots sharing following 18 elements** _org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(FilteredDocIdSetIterator.java:60)_ _org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:257)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)_ _org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)_ _org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)_ _org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)_ _org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:117)_ _org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)_ _org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)_ _org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)_ _org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ **3/10 snapshots sharing following 12 elements** _org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)_ _org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)_ _org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:117)_ _org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)_ _org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)_ _org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)_ _org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)_ _org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)_ _java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)_ _java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)_ _java.lang.Thread.run(Thread.java:745)_ * * * Our cluster is based on two AWS EC2 **r3.large** instances with **15GB memory and SSDs** for the storage. We **activated mlockall** (we checked also that it is active) and set our **heap space to 10 GB** (we also tried exactly 50% with 7 GB, no change). The cluster have 4 indexes with each 1 replica and 5 shards for later scaling requirements. We monitor and log all of our queries and respond times and there isn't a really spiky one which causes problems. Another question is why the cluster don't use all of the memory or at least the defined heap size of 10GB... 80% of our queries run against the whole dataset and there is no option to change it. Here are some general stats: **Nodes:** 2 **Indices:** 5 **Shards:** 32 **Data:** 28.69 GB **CPU:** 362% **Memory:** 3.62 GB (used) / 19.97 GB (available) **Up time:** 16.4 h **Version:** 1.5.2 Any suggestions? We don't really find the problem... Thanks a lot! Steph ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 17, 2015, 12:27am UTC](https://discuss.elastic.co/t/massive-performance-issues-on-our-production-cluster/2785/2 "2015-06-17T00:27:41Z")
Have you checked your ES logs to see if it is GC? What sort of queries do you run? ---
**Author:** ![salat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/salat/32/690_2.png) [@salat](https://discuss.elastic.co/u/salat)\ **Post date:** [June 19, 2015, 12:52pm UTC](https://discuss.elastic.co/t/massive-performance-issues-on-our-production-cluster/2785/3 "2015-06-19T12:52:07Z")
No problems with GC. Mostly aggregation queries... ---
**Author:** ![Harlin\_ES](https://avatars.discourse-cdn.com/v4/letter/h/977dab/32.png) [@Harlin\_ES](https://discuss.elastic.co/u/Harlin_ES)\ **Post date:** [June 19, 2015, 5:00pm UTC](https://discuss.elastic.co/t/massive-performance-issues-on-our-production-cluster/2785/4 "2015-06-19T17:00:40Z")
Aggregation queries can be very expensive, similar to sorting they use field data. Depending on how you store your field data this can cause problems. By default ES keeps field data inside Lucene inverted indices. Which mean every time you execute a new aggregation, Lucene has to un-invert said indices in order to pull out the field data, load it into memory, then perform the aggregation. There are a couple options to speed this up: 1. Use Doc Values -\> [https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html) 2. Eagerly load field data -\> [https://www.elastic.co/guide/en/elasticsearch/guide/current/preload-fielddata.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/preload-fielddata.html) ---
**Author:** ![salat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/salat/32/690_2.png) [@salat](https://discuss.elastic.co/u/salat)\ **Post date:** [June 20, 2015, 9:48am UTC](https://discuss.elastic.co/t/massive-performance-issues-on-our-production-cluster/2785/5 "2015-06-20T09:48:49Z")
These are great tips, thank you very much! I wasn't aware of the future perspective of Doc Values and dive a little bit into it and come back with the results. Best Steph ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:06am UTC](https://discuss.elastic.co/t/massive-performance-issues-on-our-production-cluster/2785/6 "2017-07-06T00:06:19Z")