# 100% CPU utilization on one of the data node, ES version 2.3

**URL:** <https://discuss.elastic.co/t/100-cpu-utilization-on-one-of-the-data-node-es-version-2-3/151611>\
**Category:** Elasticsearch\
**Created:** [October 9, 2018, 10:41am UTC](https://discuss.elastic.co/t/100-cpu-utilization-on-one-of-the-data-node-es-version-2-3/151611 "2018-10-09T10:41:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nagarjunadn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nagarjunadn/32/16115_2.png) [@nagarjunadn](https://discuss.elastic.co/u/nagarjunadn)\
**Post date:** [October 9, 2018, 10:41am UTC](https://discuss.elastic.co/t/100-cpu-utilization-on-one-of-the-data-node-es-version-2-3/151611/1 "2018-10-09T10:41:14Z")

</div>

Hello,

I am facing problem where one of my data nodes starts having 100% CPU usage while others don't(Elasticsearch version is 2.3).  
Restarting elastic process solves the issue for a while.

I have referred to multiple cases here but nothing helped me.

Below are the hot threads:  
100.3% (501.6ms out of 500ms) cpu usage by thread 'elasticsearch[ip-ip.ap-southeast-1.compute.internal][[**testindexc**][1]: Lucene Merge Thread #326]'  
4/10 snapshots sharing following 14 elements  
org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer$4$1.next(Lucene54DocValuesConsumer.java:754)  
org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer$4$1.next(Lucene54DocValuesConsumer.java:745)  
org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.addNumericField(Lucene54DocValuesConsumer.java:243)  
org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.addSortedNumericField(Lucene54DocValuesConsumer.java:634)  
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedNumericField(PerFieldDocValuesFormat.java:126)  
org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:417)  
org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:236)  
org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:150)  
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)  
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)  
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)  
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)  
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)  
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)  
2/10 snapshots sharing following 13 elements  
org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer$4$1.next(Lucene54DocValuesConsumer.java:745)  
org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.addNumericField(Lucene54DocValuesConsumer.java:243)  
org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.addSortedNumericField(Lucene54DocValuesConsumer.java:634)  
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedNumericField(PerFieldDocValuesFormat.java:126)  
org.apache.lucene.codecs.DocValuesConsumer.mergeSortedNumericField(DocValuesConsumer.java:417)  
org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:236)  
org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:150)  
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)  
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4075)  
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)  
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)  
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:94)  
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)

Cluster details:  
{  
"cluster\_name" : "prod-elasticsearch-cluster",  
"status" : "green",  
"timed\_out" : false,  
"number\_of\_nodes" : 17,  
"number\_of\_data\_nodes" : 11,  
"active\_primary\_shards" : 21,  
"active\_shards" : 117,  
"relocating\_shards" : 0,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 0,  
"delayed\_unassigned\_shards" : 0,  
"number\_of\_pending\_tasks" : 0,  
"number\_of\_in\_flight\_fetch" : 0,  
"task\_max\_waiting\_in\_queue\_millis" : 0,  
"active\_shards\_percent\_as\_number" : 100.0  
}

Master Nodes: 3  
Ingestion Nodes: 3  
Data Nodes: 11

The index **testindexc** which showing in hot\_threads have 3 shards and 9 replicas, total size is 25GB. Kindly suggest best possible numbers(shards vs replica)

Please help. Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 6, 2018, 10:50am UTC](https://discuss.elastic.co/t/100-cpu-utilization-on-one-of-the-data-node-es-version-2-3/151611/2 "2018-11-06T10:50:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.