Hi,
Having to test the performance of the ES with some load testing it reaches
100% CPU utilization and following is hot threads :
ES configuration
ES_MIN_MEM 256m
ES_MAX_MEM 1g
Load testing configuration
From 100 threads per second ramping up to 400 threads per second over
period of 60 second and it starts failing we reaches 300+ threads per
seconds
Hot threads
:/elasticsearch/logs ]$ curl -XGET
'http://localhost:9200//_nodes/hot_threads?pretty'
:: [Mesmero][if9uerQWRVGmpD3iq0ELYg][2905f3223ee8][inet[/172.17.0.62:9300]]
37.6% (188ms out of 500ms) cpu usage by thread
'elasticsearch[Mesmero][search][T#1]'
2/10 snapshots sharing following 6 elements
java.lang.System.currentTimeMillis(Native Method)
org.elasticsearch.action.search.type.AbstractAsyncAction.buildTookInMillis(AbstractAsyncAction.java:46)
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:150
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 18 elements
org.apache.lucene.util.AttributeSource.clearAttributes(AttributeSource.java:279)
org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:194)
org.apache.lucene.analysis.standard.StandardFilter.incrementToken(StandardFilter.java:57)
org.apache.lucene.analysis.en.EnglishPossessiveFilter.incrementToken(EnglishPossessiveFilter.java:57)
org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:62)
org.apache.lucene.analysis.util.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:90)
org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:91)
org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:218)
org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
unique snapshot
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:101)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:135)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:354)
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:335)
org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110)
org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:228)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:156)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
34.0% (169.9ms out of 500ms) cpu usage by thread
'elasticsearch[Mesmero][search][T#3]'
2/10 snapshots sharing following 13 elements
java.util.IdentityHashMap.entrySet(IdentityHashMap.java:1098)
org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:251)
org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:230)
org.elasticsearch.search.internal.InternalSearchResponse.writeTo(InternalSearchResponse.java:138)
org.elasticsearch.action.search.SearchResponse.writeTo(SearchResponse.java:262)
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:91)
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:68)
org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:150
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 20 elements
org.elasticsearch.index.query.ScriptFilterParser$ScriptFilter$ScriptDocSet.matchDoc(ScriptFilterParser.java:190)
org.elasticsearch.common.lucene.docset.MatchDocIdSet.get(MatchDocIdSet.java:67)
org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 12 elements
org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:96)
org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:218)
org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
22.8% (114.1ms out of 500ms) cpu usage by thread
'elasticsearch[Mesmero][search][T#4]'
5/10 snapshots sharing following 36 elements
java.lang.Throwable.fillInStackTrace(Native Method)
java.lang.Throwable.fillInStackTrace(Throwable.java:783)
java.lang.Throwable.(Throwable.java:250)
java.lang.Exception.(Exception.java:54)
java.lang.RuntimeException.(RuntimeException.java:51)
org.antlr.v4.runtime.RecognitionException.(RecognitionException.java:65)
org.antlr.v4.runtime.NoViableAltException.(NoViableAltException.java:70)
org.antlr.v4.runtime.atn.ParserATNSimulator.noViableAlt(ParserATNSimulator.java:1776)
org.antlr.v4.runtime.atn.ParserATNSimulator.execATN(ParserATNSimulator.java:496)
org.antlr.v4.runtime.atn.ParserATNSimulator.adaptivePredict(ParserATNSimulator.java:423)
com.test.SecurityExpressionParser.security_rule(SecurityExpressionParser.java:284)
com.test.SecurityExpressionParser.security_rule(SecurityExpressionParser.java:310)
com.test.SecurityExpressionParser.security_rule(SecurityExpressionParser.java:227)
com.test.ExpressionManager.compile(ExpressionManager.java:134)
com.test.ExpressionManager.valiateExpression(ExpressionManager.java:293)
com.test.VisibilityFilter.run(VisibilityFilter.java:107)
org.elasticsearch.index.query.ScriptFilterParser$ScriptFilter$ScriptDocSet.matchDoc(ScriptFilterParser.java:190)
org.elasticsearch.common.lucene.docset.MatchDocIdSet.get(MatchDocIdSet.java:67)
org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:60)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 7 elements
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
unique snapshot
java.lang.System.nanoTime(Native Method)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:487)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Anyone have any idea ?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5d52ccbd-8777-47c9-855c-6a96d9e0a944%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.