Elasticsearch High CPU usage

kkkk7 · February 10, 2024, 5:35pm

Hello

I'm using version 7.3.2 (this is a project since 2019 so a bit old version)
After a years pass by my elastic stack CPU usage very high so we decide extends the CPU core but seem like the usage will growth bigger and bigger

My use case is just website search with _all on 9 index and total data not more than 75k row

Are these cause of my query ? or my elastic version is too old ? or this is the normal CPU usage of Elasticsearch

thanks for your help!

Here is some info:

Hot Threads

::: {el1.xxx}{4Lg53tv3SpG5aqxJxeNIsQ}{7FyEC9R3S6uFQ2aO6ntZKw}{172.31.1.88}{172.31.1.88:9300}{dim}{ml.machine_memory=33731186688, xpack.installed=true, ml.max_open_jobs=20}
   Hot threads at 2024-02-10T17:20:42.703Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   33.9% (169.3ms out of 500ms) cpu usage by thread 'elasticsearch[el1.xxx][http_server_worker][T#2]'
     2/10 snapshots sharing following 7 elements
       io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1073)
       io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
       io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
     3/10 snapshots sharing following 9 elements
       java.base@12.0.2/sun.nio.ch.EPoll.wait(Native Method)
       java.base@12.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
       io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:791)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:439)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
   
   28.3% (141.4ms out of 500ms) cpu usage by thread 'elasticsearch[el1.xxx][http_server_worker][T#4]'
     2/10 snapshots sharing following 19 elements
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
       io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
       io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
       io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
       io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
       io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
       io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)
       io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
     8/10 snapshots sharing following 9 elements
       java.base@12.0.2/sun.nio.ch.EPoll.wait(Native Method)
       java.base@12.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
       io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:791)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:439)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
   
   26.9% (134.2ms out of 500ms) cpu usage by thread 'elasticsearch[el1.xxx][http_server_worker][T#1]'
     2/10 snapshots sharing following 7 elements
       io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
       io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)
       io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
     3/10 snapshots sharing following 9 elements
       java.base@12.0.2/sun.nio.ch.EPoll.wait(Native Method)
       java.base@12.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
       io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:791)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:439)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)

leandrojmp · February 10, 2024, 7:12pm

Hello,

Where is the high CPU usage? The screenshot you shared does not show any high CPU usage, the highest load is the 15 min average, which is 1.75, which is low for a 8 core CPU machine.

kkkk7 · February 11, 2024, 2:54am

My apologies that capture at the night low traffic time, Actually we want to reduce our spec, Last month it's peak to 100% and we have try reset all index and it's reduce to maximum at 80% in a day time but we afraid that It will growth up again in the future.

system · March 10, 2024, 2:55am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ES High cpu issues Elasticsearch	11	1062	September 6, 2018
High CPU consumption Elasticsearch	8	8632	July 5, 2017
Elasticsearch high CPU usage and stalling Elasticsearch	2	2845	July 5, 2017
High CPU usage when idle Elasticsearch	6	2300	July 5, 2017
My Elasticsearch is running at very high CPU (constantly 99%) - Need help understanding hot_threads Elasticsearch	2	1024	July 6, 2017

Elasticsearch High CPU usage

Related topics