We are getting high cpu alerts on ES nodes every day first I though like beacause of heap size, I changed it to 31g coz my machines are 64g but it does not resolved the cpu issues.
What load is the cluster under? How much data do you have in the cluster? What appears to be causing the high CPU usage if you call the node hot threads API?
This is 17 node cluster and we have 4852 indices and 18330 shards.
Which version of Elasticsearch are you using? What does the hot threads show when a node is busy?
54.4% (271.9ms out of 500ms) cpu usage by thread 'elasticsearch[search][T#17]'
10/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
52.4% (262ms out of 500ms) cpu usage by thread 'elasticsearch[][search][T#14]'
10/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
51.3% (256.3ms out of 500ms) cpu usage by thread 'elasticsearch[][search][T#29]'
3/10 snapshots sharing following 17 elements
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:221)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:410)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryScrollTransportHandler.messageReceived(SearchServiceTransportAction.java:384)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryScrollTransportHandler.messageReceived(SearchServiceTransportAction.java:381)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
We are using ES 2.3 version.
It seems to be busy with search. What type of queries are you running? What is the use case?
If i use coordinating node on my cluster, Is that useful for this problem?
I can not tell because I do not know what the problem is.
Queries are related to , we are running anti-virus app so we ll collect the samples like urls, hos,domain, ips.
What type of queries are you running? Can you provide some samples? Are you using wildcard queries, scripted fields or complex scoring?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.