# High CPU load (3-4) and 50% CPU utilization

**URL:** <https://discuss.elastic.co/t/high-cpu-load-3-4-and-50-cpu-utilization/25390>\
**Category:** Elasticsearch\
**Created:** [July 12, 2015, 2:38pm UTC](https://discuss.elastic.co/t/high-cpu-load-3-4-and-50-cpu-utilization/25390 "2015-07-12T14:38:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nir\_Reuveny](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@Nir\_Reuveny](https://discuss.elastic.co/u/Nir_Reuveny)\
**Post date:** [July 12, 2015, 2:38pm UTC](https://discuss.elastic.co/t/high-cpu-load-3-4-and-50-cpu-utilization/25390/1 "2015-07-12T14:38:29Z")

</div>

Hi,

We're running a test bulk indexing with about 10-15K documents per second on 2 node cluster of c4.xlarge instances.  
The bulk indexing is the only operations that we're running on the ES. (We use the latest 1.6.0 ES version)  
We have 8 shards for the index (4 on each node)

We noticed that the CPU utilization doesn't go above the 50% area (the instance has 4 vCPU's), but the CPU load is very high ~4 which means 'CPU bottleneck' from what I understand... why is that? why doesn't it use all available CPU?  
Is this normal?

What settings can we improve/check to lower the CPU load?  
Disks doesn't seem to be the bottleneck here, as the IOPS used are not reaching the disk limits.

---

<div class="post-metadata">

**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\
**Post date:** [July 12, 2015, 3:08pm UTC](https://discuss.elastic.co/t/high-cpu-load-3-4-and-50-cpu-utilization/25390/2 "2015-07-12T15:08:41Z")

</div>

what hot thread said? [https://www.elastic.co/guide/en/elasticsearch/reference/1.6/cluster-nodes-hot-threads.html](https://www.elastic.co/guide/en/elasticsearch/reference/1.6/cluster-nodes-hot-threads.html)

you are using vcpu, any cpu cycle stealing? you can see in the command top.

---

<div class="post-metadata">

**Author:** ![Nir\_Reuveny](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@Nir\_Reuveny](https://discuss.elastic.co/u/Nir_Reuveny)\
**Post date:** [July 12, 2015, 3:24pm UTC](https://discuss.elastic.co/t/high-cpu-load-3-4-and-50-cpu-utilization/25390/3 "2015-07-12T15:24:58Z")

</div>

No CPU steal, and idle is ~50%

Below is the hot-threads output:

::: [elasticsearch1.localdomain][CXZ9TSGmQwu7GVJi7oMRWw][elasticsearch1][inet[/10.0.0.174:9300]]{aws\_availability\_zone=us-east-1c, max\_local\_storage\_nodes=1}  
Hot threads at 2015-07-12T15:18:43.899Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

```
4.3% (21.4ms out of 500ms) cpu usage by thread 'elasticsearch[elasticsearch1.localdomain][refresh][T#4]'
 10/10 snapshots sharing following 31 elements
   sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
   sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
   sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
   sun.nio.ch.IOUtil.read(IOUtil.java:197)
   sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:699)
   sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
   org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
   org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:160)
   org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
   org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:280)
   org.apache.lucene.store.Directory.copy(Directory.java:194)
   org.apache.lucene.store.TrackingDirectoryWrapper.copy(TrackingDirectoryWrapper.java:50)
   org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4785)
   org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:511)
   org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:478)
   org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:521)
   org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:632)
   org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:410)
   org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:297)
   org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:272)
   org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
   org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:171)
   org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:118)
   org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
   org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
   org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
   org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:581)
   org.elasticsearch.index.engine.InternalEngine$1.run(InternalEngine.java:400)
   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   java.lang.Thread.run(Thread.java:745)

3.7% (18.5ms out of 500ms) cpu usage by thread 'elasticsearch[elasticsearch1.localdomain][bulk][T#2]'
 10/10 snapshots sharing following 32 elements
   sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
   sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
   sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
   sun.nio.ch.IOUtil.read(IOUtil.java:197)
   sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:699)
   sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
   org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
   org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:160)
   org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
   org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:280)
   org.apache.lucene.store.Directory.copy(Directory.java:194)
   org.apache.lucene.store.TrackingDirectoryWrapper.copy(TrackingDirectoryWrapper.java:50)
   org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4785)
   org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:511)
   org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:478)
   org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:521)
   org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:364)
   org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1526)
   org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1252)
   org.elasticsearch.index.engine.InternalEngine.innerCreateNoLock(InternalEngine.java:356)
   org.elasticsearch.index.engine.InternalEngine.innerCreate(InternalEngine.java:298)
   org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:269)
   org.elasticsearch.index.shard.IndexShard.create(IndexShard.java:483)
   org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:423)
   org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:148)
   org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
   org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
   org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   java.lang.Thread.run(Thread.java:745)

2.9% (14.5ms out of 500ms) cpu usage by thread 'elasticsearch[elasticsearch1.localdomain][refresh][T#1]'
 10/10 snapshots sharing following 31 elements
   sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
   sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
   sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
   sun.nio.ch.IOUtil.read(IOUtil.java:197)
   sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:699)
   sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684)
   org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
   org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:160)
   org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
   org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:280)
   org.apache.lucene.store.Directory.copy(Directory.java:194)
   org.apache.lucene.store.TrackingDirectoryWrapper.copy(TrackingDirectoryWrapper.java:50)
   org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4785)
   org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:511)
   org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:478)
   org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:521)
   org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:632)
   org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:410)
   org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:297)
   org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:272)
   org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
   org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:171)
   org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:118)
   org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
   org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
   org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
   org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:581)
   org.elasticsearch.index.engine.InternalEngine$1.run(InternalEngine.java:400)
   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   java.lang.Thread.run(Thread.java:745)

```

::: [elasticsearch2.localdomain][Xl5RhqNNT1eyskbArZof2w][elasticsearch2][inet[/10.0.0.189:9300]]{aws\_availability\_zone=us-east-1c, max\_local\_storage\_nodes=1}  
Hot threads at 2015-07-12T15:18:43.898Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

54.0% (270ms out of 500ms) cpu usage by thread 'elasticsearch[elasticsearch2.localdomain][[lld-test-index1][7]: Lucene Merge Thread #32]'  
3/10 snapshots sharing following 14 elements  
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:149)  
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:170)  
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:243)  
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:161)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:236)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:166)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:388)  
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:332)  
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:100)  
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4223)  
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3811)  
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)  
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)  
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)  
7/10 snapshots sharing following 7 elements  
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:332)  
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:100)  
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4223)  
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3811)  
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)  
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)  
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)

27.8% (138.8ms out of 500ms) cpu usage by thread 'elasticsearch[elasticsearch2.localdomain][[lld-test-index1][3]: Lucene Merge Thread #32]'  
5/10 snapshots sharing following 20 elements  
java.lang.Thread.sleep(Native Method)  
java.lang.Thread.sleep(Thread.java:340)  
org.apache.lucene.store.RateLimiter$SimpleRateLimiter.pause(RateLimiter.java:151)  
org.apache.lucene.store.RateLimitedFSDirectory$RateLimiterWrapper.pause(RateLimitedFSDirectory.java:96)  
org.apache.lucene.store.RateLimitedIndexOutput.checkRate(RateLimitedIndexOutput.java:76)  
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:70)  
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:157)  
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:170)  
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:243)  
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:161)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:236)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:166)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:388)  
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:332)  
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:100)  
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4223)  
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3811)  
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)  
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)  
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)  
3/10 snapshots sharing following 19 elements  
java.lang.Thread.sleep(Native Method)  
java.lang.Thread.sleep(Thread.java:340)  
org.apache.lucene.store.RateLimiter$SimpleRateLimiter.pause(RateLimiter.java:151)  
org.apache.lucene.store.RateLimitedFSDirectory$RateLimiterWrapper.pause(RateLimitedFSDirectory.java:96)  
org.apache.lucene.store.RateLimitedIndexOutput.checkRate(RateLimitedIndexOutput.java:76)  
org.apache.lucene.store.RateLimitedIndexOutput.writeByte(RateLimitedIndexOutput.java:63)  
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:175)  
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:243)  
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:161)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:236)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:166)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:388)  
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:332)  
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:100)  
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4223)  
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3811)  
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)  
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)  
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)  
2/10 snapshots sharing following 13 elements  
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:170)  
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:243)  
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:161)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:236)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:166)  
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:388)  
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:332)  
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:100)  
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4223)  
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3811)  
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)  
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)  
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)

---

<div class="post-metadata">

**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\
**Post date:** [July 12, 2015, 3:47pm UTC](https://discuss.elastic.co/t/high-cpu-load-3-4-and-50-cpu-utilization/25390/4 "2015-07-12T15:47:01Z")

</div>

looks like cpu is busy merging lucene segments. read segment merging background here [https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html) before you start to do some tuning.

hth

---

<div class="post-metadata">

**Author:** ![Nir\_Reuveny](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@Nir\_Reuveny](https://discuss.elastic.co/u/Nir_Reuveny)\
**Post date:** [July 12, 2015, 3:49pm UTC](https://discuss.elastic.co/t/high-cpu-load-3-4-and-50-cpu-utilization/25390/5 "2015-07-12T15:49:06Z")

</div>

But why doesn't it use more than 50% CPU utilization if the CPU load is ~4?  
Any ideas?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:02am UTC](https://discuss.elastic.co/t/high-cpu-load-3-4-and-50-cpu-utilization/25390/6 "2017-07-06T00:02:03Z")

</div>