Elastic Search CPU Utilization.. on one cluster node Version1.7.2

Elastic Stack Elasticsearch
1

image

Hi Team,

I am observing that there is a high CPU utilization on one my culster server. This has been regular.. I am helpless to my team.. I request you to please let me know what i need to do on this regards..

I restart elastic service on that node but no use.. I see the error as below when i check for hotthreads

::: [Prod-Node-Three-25][zP4lrdagQeSWxm6204N30A][pro-phvapp04][inet[/172.23.12.25:9300]]
Hot threads at 2019-04-25T05:28:52.758Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

103.1% (515.6ms out of 500ms) cpu usage by thread 'elasticsearch[Prod-Node-Three-25][search][T#6]'
5/10 snapshots sharing following 14 elements
org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:454)
org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:114)
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.search.fetch.matchedqueries.MatchedQueriesFetchSubPhase.addMatchedQueries(MatchedQueriesFetchSubPhase.java:123)
org.elasticsearch.search.fetch.matchedqueries.MatchedQueriesFetchSubPhase.hitExecute(MatchedQueriesFetchSubPhase.java:80)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:516)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
5/10 snapshots sharing following 9 elements
org.elasticsearch.search.fetch.matchedqueries.MatchedQueriesFetchSubPhase.hitExecute(MatchedQueriesFetchSubPhase.java:80)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:516)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)

2

It looks like there as a potentially expensive query running. How many nodes do you have in the cluster?

1.7.2 is also a very old version, which has been End-Of-Life for a very long time. I would therefore recommend you consider upgrading.

3

Thank you for the support. We will take it for the upgrade of this to latest version..
It has 2 nodes.

The node which consumes CPU is always a child node. Because of this consumption, The queries are effected on the master node.

How can i query, What expanse queries got executed?.

Till we upgrade it, Can you suggest us the alternative/remedy.

Many Thanks,
Kamesh

4

You might try enabling and tuning the slow log, but I have not used version 1.7 in years so have not got any other suggestions.

5

Thank you Christian for the quick response
Will check on you suggestion