High I have 2 nodes in Elasticsearch cluster. 1 master and 1 data node. The master node is also a data node
I have 2 logstash instances running and one filebeat on separate machines.
The load is not too much just a modest 100-200 req per sec.
But I am facing some performance issues. On Elasticsearch the CPU usage is too high 150-250%. If I try to get the hot threads then I get the following output
::: {node124}{lo4V14MESv2jDG5oK3bzwQ}{v4osXlRkQC2WDONT1fpNog}{-.-.-.-}{-.-.-.-:9300}{cdfhilrstw}{ml.machine_memory=66995937280, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1073741824}
Hot threads at 2022-06-29T07:40:52.008Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=0.6%, other=99.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[node124][transport_worker][T#17]'
3/10 snapshots sharing following 3 elements
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
java.base@18/java.lang.Thread.run(Thread.java:833)
::: {node201}{I5WCg6avQV6S_ebq-z3DCA}{4nMCyTuPSdWDUisMvIUpkQ}{0.0.0.0}{0.0.0.0:9300}{cdfhilmrstw}{ml.machine_memory=67350491136, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=1073741824, transform.node=true}
Hot threads at 2022-06-29T07:29:45.081Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=1.6%, other=98.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[node201][transport_worker][T#3]'
2/10 snapshots sharing following 3 elements
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
java.base@18/java.lang.Thread.run(Thread.java:833)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.