Elasticsearch High CPU usage

Elastic Stack Elasticsearch
1

Hello

I'm using version 7.3.2 (this is a project since 2019 so a bit old version)
After a years pass by my elastic stack CPU usage very high so we decide extends the CPU core but seem like the usage will growth bigger and bigger

My use case is just website search with _all on 9 index and total data not more than 75k row

Are these cause of my query ? or my elastic version is too old ? or this is the normal CPU usage of Elasticsearch

thanks for your help!

Here is some info:

Hot Threads

::: {el1.xxx}{4Lg53tv3SpG5aqxJxeNIsQ}{7FyEC9R3S6uFQ2aO6ntZKw}{172.31.1.88}{172.31.1.88:9300}{dim}{ml.machine_memory=33731186688, xpack.installed=true, ml.max_open_jobs=20}
   Hot threads at 2024-02-10T17:20:42.703Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   33.9% (169.3ms out of 500ms) cpu usage by thread 'elasticsearch[el1.xxx][http_server_worker][T#2]'
     2/10 snapshots sharing following 7 elements
       io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1073)
       io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
       io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
     3/10 snapshots sharing following 9 elements
       java.base@12.0.2/sun.nio.ch.EPoll.wait(Native Method)
       java.base@12.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
       io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:791)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:439)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
   
   28.3% (141.4ms out of 500ms) cpu usage by thread 'elasticsearch[el1.xxx][http_server_worker][T#4]'
     2/10 snapshots sharing following 19 elements
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
       io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
       io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
       io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
       io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
       io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
       io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)
       io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
     8/10 snapshots sharing following 9 elements
       java.base@12.0.2/sun.nio.ch.EPoll.wait(Native Method)
       java.base@12.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
       io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:791)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:439)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
   
   26.9% (134.2ms out of 500ms) cpu usage by thread 'elasticsearch[el1.xxx][http_server_worker][T#1]'
     2/10 snapshots sharing following 7 elements
       io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
       io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)
       io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)
     3/10 snapshots sharing following 9 elements
       java.base@12.0.2/sun.nio.ch.EPoll.wait(Native Method)
       java.base@12.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
       java.base@12.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
       io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:791)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:439)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@12.0.2/java.lang.Thread.run(Thread.java:835)

image
image1920×860 441 KB

2

Hello,

Where is the high CPU usage? The screenshot you shared does not show any high CPU usage, the highest load is the 15 min average, which is 1.75, which is low for a 8 core CPU machine.

1 Like
3

My apologies that capture at the night low traffic time, Actually we want to reduce our spec, Last month it's peak to 100% and we have try reset all index and it's reduce to maximum at 80% in a day time but we afraid that It will growth up again in the future.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.