Elasticsearch High CPU high load when searching

Elastic Stack Elasticsearch
1

Hi,everyone
I'm new to es ,i got a probem with es and tried a lot but still don't know
how to change the situation ,so i would be gratitude if you can give me
some advice about better performance of es searching

I build a es cluster with 2 nodes, in order to analyse logs for develop engineers
,like some 5XX,4XX status and so on. It's running good ,but when i open
kibana (only one user) which will do some queries to es, then the 2 servers
will perform high cpu load. when i close kibana ,cpu load will be nomal
again.

there are some details about the problem below:

1.marvel basic monitor
https://plus.google.com/107610728578662862458/posts/GM8jjjX1HTi

2.gc monitor
https://plus.google.com/107610728578662862458/posts/PW7LCeDjemq

3.hot thread
https://docs.google.com/a/widget-inc.com/document/d/199CS0YHrAOWIbrNZssjpArOH9E0F11wHwIrO9XjmT-U/edit?usp=sharing

My server basic information:

RAM: 48G
cpu cores :4
centos 6.5 kernel version 3.8.13

jdk : jdk1.7.0_67
es version : 1.1.1

the work i have done:

when i google the problem, i got some possible solution

1.the leap-second
since my kernel is 3.8.13, so the kernel had solved this problem

I'm sorry for my bad english.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cb463ea7-88d2-417e-b3d9-07256c2e3b6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Your second two links do not work.

Can you add another node, or close some old indexes?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 4 September 2014 20:18, 王星龙 xinglong.wang@upai.com wrote:

Hi,everyone
I'm new to es ,i got a probem with es and tried a lot but still don't know
how to change the situation ,so i would be gratitude if you can give me
some advice about better performance of es searching

I build a es cluster with 2 nodes, in order to analyse logs for develop engineers
,like some 5XX,4XX status and so on. It's running good ,but when i open
kibana (only one user) which will do some queries to es, then the 2 servers
will perform high cpu load. when i close kibana ,cpu load will be nomal
again.

there are some details about the problem below:

1.marvel basic monitor
https://plus.google.com/107610728578662862458/posts/GM8jjjX1HTi

2.gc monitor
https://plus.google.com/107610728578662862458/posts/PW7LCeDjemq

3.hot thread
https://docs.google.com/a/widget-inc.com/document/d/199CS0YHrAOWIbrNZssjpArOH9E0F11wHwIrO9XjmT-U/edit?usp=sharing

My server basic information:

RAM: 48G
cpu cores :4
centos 6.5 kernel version 3.8.13

jdk : jdk1.7.0_67
es version : 1.1.1

the work i have done:

when i google the problem, i got some possible solution

1.the leap-second
since my kernel is 3.8.13, so the kernel had solved this problem

I'm sorry for my bad english.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cb463ea7-88d2-417e-b3d9-07256c2e3b6a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cb463ea7-88d2-417e-b3d9-07256c2e3b6a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bWUfahJzfVqM4D3jw494kBDXp7wRJNEDNtAs7UtfFZrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

gc-monitor pic :

hot-thread:

34.2% (171ms out of 500ms) cpu usage by thread
'elasticsearch[node203][search][T#10]'

2/10 snapshots sharing following 23 elements

  org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:66)

  org.apache.lucene.search.DisjunctionScorer.advance(DisjunctionScorer.java:150)

  org.apache.lucene.search.ReqExclScorer.toNonExcluded(ReqExclScorer.java:79)

  org.apache.lucene.search.ReqExclScorer.nextDoc(ReqExclScorer.java:58)

  org.apache.lucene.search.BooleanScorer2.nextDoc(BooleanScorer2.java:311)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:130)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

2/10 snapshots sharing following 23 elements

  org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:56)

  org.apache.lucene.search.DisjunctionScorer.advance(DisjunctionScorer.java:150)

  org.apache.lucene.search.ReqExclScorer.toNonExcluded(ReqExclScorer.java:79)

  org.apache.lucene.search.ReqExclScorer.nextDoc(ReqExclScorer.java:58)

  org.apache.lucene.search.BooleanScorer2.nextDoc(BooleanScorer2.java:311)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:130)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

2/10 snapshots sharing following 20 elements

  org.apache.lucene.search.ReqExclScorer.nextDoc(ReqExclScorer.java:58)

  org.apache.lucene.search.BooleanScorer2.nextDoc(BooleanScorer2.java:311)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:130)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

4/10 snapshots sharing following 19 elements

  org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:61)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:131)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

25.1% (125.7ms out of 500ms) cpu usage by thread
'elasticsearch[node203][search][T#7]'

10/10 snapshots sharing following 16 elements

  org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:297)

  org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612)

  org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

24.3% (121.6ms out of 500ms) cpu usage by thread
'elasticsearch[node203][search][T#13]'

2/10 snapshots sharing following 23 elements

  org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:61)

  org.apache.lucene.search.DisjunctionScorer.advance(DisjunctionScorer.java:150)

  org.apache.lucene.search.ReqExclScorer.toNonExcluded(ReqExclScorer.java:79)

  org.apache.lucene.search.ReqExclScorer.nextDoc(ReqExclScorer.java:58)

  org.apache.lucene.search.BooleanScorer2.nextDoc(BooleanScorer2.java:311)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:130)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

3/10 snapshots sharing following 22 elements

  org.apache.lucene.search.DisjunctionScorer.advance(DisjunctionScorer.java:158)

  org.apache.lucene.search.ReqExclScorer.toNonExcluded(ReqExclScorer.java:79)

  org.apache.lucene.search.ReqExclScorer.nextDoc(ReqExclScorer.java:58)

  org.apache.lucene.search.BooleanScorer2.nextDoc(BooleanScorer2.java:311)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:130)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

2/10 snapshots sharing following 24 elements

  org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(FilteredDocIdSetIterator.java:60)

  org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:196)

  org.apache.lucene.search.ConjunctionScorer.nextDoc(ConjunctionScorer.java:99)

  org.apache.lucene.search.ReqExclScorer.toNonExcluded(ReqExclScorer.java:88)

  org.apache.lucene.search.ReqExclScorer.nextDoc(ReqExclScorer.java:58)

  org.apache.lucene.search.BooleanScorer2.nextDoc(BooleanScorer2.java:311)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:130)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

2/10 snapshots sharing following 19 elements

  org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:61)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:131)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

unique snapshot

  org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:56)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:131)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

:::
[node202][XRa3t0ZzRl6eiKRwcKpLsg][UPWEB-PHP-043][inet[192.168.1.43/192.168.1.43:9300]]{master=true}

60.8% (303.9ms out of 500ms) cpu usage by thread
'elasticsearch[node202][search][T#1]'

4/10 snapshots sharing following 16 elements

  org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetExecutor$Collector.collect(TermsStringOrdinalsFacetExecutor.java:223)

  org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:61)

  org.elasticsearch.common.lucene.MultiCollector.collect(MultiCollector.java:60)

  org.apache.lucene.search.Scorer.score(Scorer.java:65)

  org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:256)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612)

  org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

2/10 snapshots sharing following 14 elements

  org.elasticsearch.common.lucene.MultiCollector.collect(MultiCollector.java:60)

  org.apache.lucene.search.Scorer.score(Scorer.java:65)

  org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:256)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612)

  org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

4/10 snapshots sharing following 14 elements

  org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:196)

  org.apache.lucene.search.Scorer.score(Scorer.java:64)

  org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:256)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612)

  org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

49.0% (245ms out of 500ms) cpu usage by thread
'elasticsearch[node202][search][T#20]'

2/10 snapshots sharing following 23 elements

  org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:56)

  org.apache.lucene.search.DisjunctionScorer.advance(DisjunctionScorer.java:150)

  org.apache.lucene.search.ReqExclScorer.toNonExcluded(ReqExclScorer.java:79)

  org.apache.lucene.search.ReqExclScorer.advance(ReqExclScorer.java:129)

  org.apache.lucene.search.BooleanScorer2.advance(BooleanScorer2.java:328)

  org.apache.lucene.search.DisjunctionScorer.advance(DisjunctionScorer.java:149)

  org.apache.lucene.search.FilteredQuery$LeapFrogScorer.score(FilteredQuery.java:256)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

8/10 snapshots sharing following 17 elements

  org.apache.lucene.search.FilteredQuery$LeapFrogScorer.score(FilteredQuery.java:256)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:116)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)

  org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)

  org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

45.6% (228.1ms out of 500ms) cpu usage by thread
'elasticsearch[node202][search][T#16]'

4/10 snapshots sharing following 16 elements

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:131)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612)

  org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

2/10 snapshots sharing following 19 elements

  org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.docID(ConstantScoreQuery.java:201)

  org.apache.lucene.search.DisjunctionSumScorer.countMatches(DisjunctionSumScorer.java:64)

  org.apache.lucene.search.DisjunctionSumScorer.afterNext(DisjunctionSumScorer.java:55)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:139)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612)

  org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

2/10 snapshots sharing following 17 elements

  org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.docID(ConstantScoreQuery.java:201)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:138)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612)

  org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

2/10 snapshots sharing following 17 elements

  org.apache.lucene.search.DisjunctionSumScorer.afterNext(DisjunctionSumScorer.java:50)

  org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:139)

  org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)

  org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)

  org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:68)

  org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)

  org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)

  org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

  org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)

  org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623)

  org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612)

  org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)

  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

  java.lang.Thread.run(Thread.java:745)

在 2014年9月4日星期四UTC+8下午6时18分01秒，王星龙写道：

Hi,everyone
I'm new to es ,i got a probem with es and tried a lot but still don't know
how to change the situation ,so i would be gratitude if you can give me
some advice about better performance of es searching

I build a es cluster with 2 nodes, in order to analyse logs for develop engineers
,like some 5XX,4XX status and so on. It's running good ,but when i open
kibana (only one user) which will do some queries to es, then the 2 servers
will perform high cpu load. when i close kibana ,cpu load will be nomal
again.

there are some details about the problem below:

1.marvel basic monitor
https://plus.google.com/107610728578662862458/posts/GM8jjjX1HTi

2.gc monitor
https://plus.google.com/107610728578662862458/posts/PW7LCeDjemq

3.hot thread
https://docs.google.com/a/widget-inc.com/document/d/199CS0YHrAOWIbrNZssjpArOH9E0F11wHwIrO9XjmT-U/edit?usp=sharing

My server basic information:

RAM: 48G
cpu cores :4
centos 6.5 kernel version 3.8.13

jdk : jdk1.7.0_67
es version : 1.1.1

the work i have done:

when i google the problem, i got some possible solution

1.the leap-second
since my kernel is 3.8.13, so the kernel had solved this problem

I'm sorry for my bad english.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5a1f7bcc-af2a-4639-bd52-b52b36445c55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

i just add the link of the second pic

在 2014年9月4日星期四UTC+8下午6时29分06秒，Mark Walkom写道：

Your second two links do not work.

Can you add another node, or close some old indexes?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 4 September 2014 20:18, 王星龙 <xinglo...@upai.com <javascript:>> wrote:

Hi,everyone
I'm new to es ,i got a probem with es and tried a lot but still don't
know how to change the situation ,so i would be gratitude if you can
give me some advice about better performance of es searching

I build a es cluster with 2 nodes, in order to analyse logs for develop engineers
,like some 5XX,4XX status and so on. It's running good ,but when i open
kibana (only one user) which will do some queries to es, then the 2 servers
will perform high cpu load. when i close kibana ,cpu load will be nomal
again.

there are some details about the problem below:

1.marvel basic monitor
https://plus.google.com/107610728578662862458/posts/GM8jjjX1HTi

2.gc monitor
https://plus.google.com/107610728578662862458/posts/PW7LCeDjemq

3.hot thread
https://docs.google.com/a/widget-inc.com/document/d/199CS0YHrAOWIbrNZssjpArOH9E0F11wHwIrO9XjmT-U/edit?usp=sharing

My server basic information:

RAM: 48G
cpu cores :4
centos 6.5 kernel version 3.8.13

jdk : jdk1.7.0_67
es version : 1.1.1

the work i have done:

when i google the problem, i got some possible solution

1.the leap-second
since my kernel is 3.8.13, so the kernel had solved this problem

I'm sorry for my bad english.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cb463ea7-88d2-417e-b3d9-07256c2e3b6a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cb463ea7-88d2-417e-b3d9-07256c2e3b6a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/80aa4dd9-9452-467d-9731-ae09b97018dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

I just add the link of the second pic .

yes ,i can add more node to my cluster, but i want to know does i can do
something to improve the performance, like gc or something else ?

what do you mean by "close the old indexes" ? do you means to delete the
old indexs? if so ,does the old indexes influence the search ? because we
always search latest log (today's log), am i right?

在 2014年9月4日星期四UTC+8下午6时29分06秒，Mark Walkom写道：

Your second two links do not work.

Can you add another node, or close some old indexes?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 4 September 2014 20:18, 王星龙 <xinglo...@upai.com <javascript:>> wrote:

Hi,everyone
I'm new to es ,i got a probem with es and tried a lot but still don't
know how to change the situation ,so i would be gratitude if you can
give me some advice about better performance of es searching

I build a es cluster with 2 nodes, in order to analyse logs for develop engineers
,like some 5XX,4XX status and so on. It's running good ,but when i open
kibana (only one user) which will do some queries to es, then the 2 servers
will perform high cpu load. when i close kibana ,cpu load will be nomal
again.

there are some details about the problem below:

1.marvel basic monitor
https://plus.google.com/107610728578662862458/posts/GM8jjjX1HTi

2.gc monitor
https://plus.google.com/107610728578662862458/posts/PW7LCeDjemq

3.hot thread
https://docs.google.com/a/widget-inc.com/document/d/199CS0YHrAOWIbrNZssjpArOH9E0F11wHwIrO9XjmT-U/edit?usp=sharing

My server basic information:

RAM: 48G
cpu cores :4
centos 6.5 kernel version 3.8.13

jdk : jdk1.7.0_67
es version : 1.1.1

the work i have done:

when i google the problem, i got some possible solution

1.the leap-second
since my kernel is 3.8.13, so the kernel had solved this problem

I'm sorry for my bad english.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cb463ea7-88d2-417e-b3d9-07256c2e3b6a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cb463ea7-88d2-417e-b3d9-07256c2e3b6a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4a602b19-c4f8-4feb-bd1a-9d2fda627bef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

I want to add cache for es request, but this function is coming to 1.4.0

在 2014年9月4日星期四UTC+8下午6时29分06秒，Mark Walkom写道：

Your second two links do not work.

Can you add another node, or close some old indexes?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 4 September 2014 20:18, 王星龙 <xinglo...@upai.com <javascript:>> wrote:

Hi,everyone
I'm new to es ,i got a probem with es and tried a lot but still don't
know how to change the situation ,so i would be gratitude if you can
give me some advice about better performance of es searching

I build a es cluster with 2 nodes, in order to analyse logs for develop engineers
,like some 5XX,4XX status and so on. It's running good ,but when i open
kibana (only one user) which will do some queries to es, then the 2 servers
will perform high cpu load. when i close kibana ,cpu load will be nomal
again.

there are some details about the problem below:

1.marvel basic monitor
https://plus.google.com/107610728578662862458/posts/GM8jjjX1HTi

2.gc monitor
https://plus.google.com/107610728578662862458/posts/PW7LCeDjemq

3.hot thread
https://docs.google.com/a/widget-inc.com/document/d/199CS0YHrAOWIbrNZssjpArOH9E0F11wHwIrO9XjmT-U/edit?usp=sharing

My server basic information:

RAM: 48G
cpu cores :4
centos 6.5 kernel version 3.8.13

jdk : jdk1.7.0_67
es version : 1.1.1

the work i have done:

when i google the problem, i got some possible solution

1.the leap-second
since my kernel is 3.8.13, so the kernel had solved this problem

I'm sorry for my bad english.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cb463ea7-88d2-417e-b3d9-07256c2e3b6a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cb463ea7-88d2-417e-b3d9-07256c2e3b6a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9dcc63a6-62aa-4691-9ca2-8493b9b8e450%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.