elasticsearch-8.5.3
No response
bundled
4.19.188-10.el7
Is it normal for Elasticsearch to consume 10% CPU even without requests after running for a period of time?
top -Hp <es_pid> :
Thread 5264 is using approximately 10% of the CPU.
jstack 5264 :
strace -p 5264 :
The screen is constantly refreshing the content above.
Does anyone know why? Thank you very much.
Please run the hot threads API when the node is consuming CPU without any requests and share the output. If you have been indexing into the cluster it is possible for merging to occur even during quiet periods and this would show up if you use the API I linked to.
I have run the command many times, and in the vast majority of cases, this is the output it produces.
{ml.machine_memory=8150421504, ml.allocated_processors=2, ml.max_jvm_size=2147483648, ml.allocated_processors_double=2.0, xpack.installed=true}
Hot threads at 2023-03-25T06:32:22.015Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
In rare cases, their output is like this.
{ml.machine_memory=8150421504, ml.allocated_processors=2, ml.max_jvm_size=2147483648, ml.allocated_processors_double=2.0, xpack.installed=true}
Hot threads at 2023-03-25T06:46:03.346Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
21.7% [cpu=21.7%, other=0.0%] (108.5ms out of 500ms) cpu usage by thread 'elasticsearch[ucloud-rez4-1][system_read][T#1]'
2/10 snapshots sharing following 24 elements
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:218)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:284)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:254)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:45)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.BulkScorer.score(BulkScorer.java:38)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:196)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:464)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:627)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:489)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService$$Lambda$7249/0x00000008021f9938.get(Unknown Source)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService$$Lambda$7250/0x00000008021f9b48.get(Unknown Source)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:892)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
2/10 snapshots sharing following 23 elements
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:187)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:218)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:284)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:254)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:45)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.BulkScorer.score(BulkScorer.java:38)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:196)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$6(SearchService.java:770)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService$$Lambda$7814/0x00000008022e6888.get(Unknown Source)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService$$Lambda$7250/0x00000008021f9b48.get(Unknown Source)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:892)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
2/10 snapshots sharing following 16 elements
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:170)
app/org.apache.lucene.core@9.4.2/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:519)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:224)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:179)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:80)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$6(SearchService.java:770)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService$$Lambda$7814/0x00000008022e6888.get(Unknown Source)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.search.SearchService$$Lambda$7250/0x00000008021f9b48.get(Unknown Source)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:892)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
4/10 snapshots sharing following 13 elements
java.base@19.0.1/jdk.internal.misc.Unsafe.park(Native Method)
java.base@19.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3744)
java.base@19.0.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3689)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616)
java.base@19.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286)
app/org.elasticsearch.server@8.5.3/org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:152)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base@19.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@19.0.1/java.lang.Thread.run(Thread.java:1589)
This thread is the same as the thread shown as occupying the CPU in top.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.