Elasticsearch high load on some nodes

Elastic Stack Elasticsearch
1

os :CentOS Linux release 7.5.1804 (Core)
java:1.8.0_275
elasticsearch:6.8.1

::: {192.168.1.3_hot}{r6gbkwdTRDSUUooaLfWZPw}{djXsTjmBS3OM1IBzw4E2hA}{192.168.1.3}{192.168.1.3:9300}{ml.machine_memory=540476571648, ml.max_open_jobs=20, xpack.installed=true, box_type=hot, ml.enabled=true}
   Hot threads at 2021-06-21T05:32:45.430Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   74.4% (372ms out of 500ms) cpu usage by thread 'elasticsearch[192.168.1.3_hot][transport_worker][T#33]'
     unique snapshot
       sun.security.provider.SHA5.lf_delta0(SHA5.java:200)
       sun.security.provider.SHA5.implCompress0(SHA5.java:243)
       sun.security.provider.SHA5.implCompress(SHA5.java:221)
       sun.security.provider.DigestBase.implCompressMultiBlock0(DigestBase.java:147)
       sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:142)
       sun.security.provider.DigestBase.engineUpdate(DigestBase.java:129)
       java.security.MessageDigestSpi.engineUpdate(MessageDigestSpi.java:114)
       java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:605)
       java.security.MessageDigest.update(MessageDigest.java:356)
       com.sun.crypto.provider.HmacCore.engineUpdate(HmacCore.java:182)
       javax.crypto.Mac.update(Mac.java:534)
       sun.security.ssl.Authenticator$MacImpl.compute(Authenticator.java:373)
       sun.security.ssl.Authenticator$TLS10Mac.compute(Authenticator.java:442)
       sun.security.ssl.SSLCipher.checkMacTags(SSLCipher.java:2185)
       sun.security.ssl.SSLCipher.checkCBCMac(SSLCipher.java:2142)
       sun.security.ssl.SSLCipher.access$1500(SSLCipher.java:54)
       sun.security.ssl.SSLCipher$T11BlockReadCipherGenerator$BlockReadCipher.decrypt(SSLCipher.java:1330)
       sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
       sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
       sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
       sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
       sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:575)
       sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:531)
       sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:398)
       sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:377)
       javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:626)
       io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295)
       io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301)
       io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
       io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
       io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
       io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
       io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
       io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
       io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
       io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
       io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
       io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
       io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
       io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
       java.lang.Thread.run(Thread.java:748)
2

What is the speecification of your cluster with respect to hardware and size? What is the full output of the cluster stats API?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.