# Elasticsearch single node is occasionally loaded very high **URL:** \ **Category:** Elasticsearch\ **Created:** [March 11, 2021, 8:27am UTC](https://discuss.elastic.co/t/elasticsearch-single-node-is-occasionally-loaded-very-high/266904 "2021-03-11T08:27:00Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![bxl](https://avatars.discourse-cdn.com/v4/letter/b/aca169/32.png) [@bxl](https://discuss.elastic.co/u/bxl)\ **Post date:** [March 11, 2021, 8:27am UTC](https://discuss.elastic.co/t/elasticsearch-single-node-is-occasionally-loaded-very-high/266904/1 "2021-03-11T08:27:00Z")
Hi, My ES cluster is used to store logs. When writing data in graylog, I found that occasionally the load of a node would suddenly increase, affecting the data written to the whole cluster. The node did not have gc, and I did not see any obvious hotlines, probably for what reason? elasticsearch:6.8.1 java:1.8 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bb447406d87d471daa853628c0bf661330984784.jpeg) ---
**Author:** ![bxl](https://avatars.discourse-cdn.com/v4/letter/b/aca169/32.png) [@bxl](https://discuss.elastic.co/u/bxl)\ **Post date:** [March 11, 2021, 11:23am UTC](https://discuss.elastic.co/t/elasticsearch-single-node-is-occasionally-loaded-very-high/266904/2 "2021-03-11T11:23:57Z")
```auto 55.3% (276.6ms out of 500ms) cpu usage by thread 'elasticsearch[10.0.2.1_hot][transport_worker][T#113]' 6/10 snapshots sharing following 34 elements org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:763) org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) -- 54.9% (274.3ms out of 500ms) cpu usage by thread 'elasticsearch[10.0.2.1_hot][transport_worker][T#110]' 6/10 snapshots sharing following 26 elements sun.security.ssl.SSLEngineImpl.isInboundDone(SSLEngineImpl.java:636) sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:551) sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:398) sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:377) javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:626) io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) -- 50.3% (251.7ms out of 500ms) cpu usage by thread 'elasticsearch[10.0.2.1_hot][transport_worker][T#128]' 10/10 snapshots sharing following 20 elements io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:797) io.netty.handler.ssl.SslHandler.flush(SslHandler.java:778) io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265) io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) io.netty.channel.AbstractChannelHandlerContext.access$1500(AbstractChannelHandlerContext.java:38) io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1152) io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1075) io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ``` ---
**Author:** ![bxl](https://avatars.discourse-cdn.com/v4/letter/b/aca169/32.png) [@bxl](https://discuss.elastic.co/u/bxl)\ **Post date:** [March 11, 2021, 1:49pm UTC](https://discuss.elastic.co/t/elasticsearch-single-node-is-occasionally-loaded-very-high/266904/3 "2021-03-11T13:49:06Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/4/b/4be098f1a4eea0acaf0eac38053c13fc8375258d.jpeg) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 11, 2021, 1:55pm UTC](https://discuss.elastic.co/t/elasticsearch-single-node-is-occasionally-loaded-very-high/266904/5 "2021-03-11T13:55:43Z")
What is the specification of your hardware? What type of storage do you have? What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cluster-stats.html)? Also please do not post images of text as they can be very hard to read and are not searchable. ---
**Author:** ![bxl](https://avatars.discourse-cdn.com/v4/letter/b/aca169/32.png) [@bxl](https://discuss.elastic.co/u/bxl)\ **Post date:** [March 12, 2021, 1:29am UTC](https://discuss.elastic.co/t/elasticsearch-single-node-is-occasionally-loaded-very-high/266904/6 "2021-03-12T01:29:11Z")
ElasticSearch：6.8.1 java：1.8.0-275 Hardware：256G RAM，40 vCore, 1.5 TB nvme0n1/SSD ---
**Author:** ![bxl](https://avatars.discourse-cdn.com/v4/letter/b/aca169/32.png) [@bxl](https://discuss.elastic.co/u/bxl)\ **Post date:** [March 12, 2021, 1:30am UTC](https://discuss.elastic.co/t/elasticsearch-single-node-is-occasionally-loaded-very-high/266904/7 "2021-03-12T01:30:48Z")
I found that the cpu sys occupancy is relatively high ```auto top - 09:23:31 up 66 days, 19:12, 2 users, load average: 58.18, 28.24, 16.51 Tasks: 735 total, 6 running, 728 sleeping, 1 stopped, 0 zombie %Cpu(s): 12.9 us, 83.7 sy, 0.0 ni, 3.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 52781283+total, 15956684 free, 23306942+used, 27878672+buff/cache KiB Swap: 0 total, 0 free, 0 used. 29171699+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 82566 elastic+ 20 0 73.7g 44.6g 126220 S 5456 8.9 630454:26 java Samples: 24K of event 'cycles:ppp', Event count (approx.): 16945917753 Overhead Shared Object Symbol 86.59% [kernel] [k] native_queued_spin_lock_slowpath 8.26% [kernel] [k] cpu_idle_poll 0.35% [kernel] [k] _raw_spin_unlock_irqrestore 0.21% [kernel] [k] compact_checklock_irqsave.isra.24 0.17% [kernel] [k] _raw_spin_lock_irqsave 0.12% perf-80906.map [.] 0x00002b16ff2e2d13 0.11% perf-80906.map [.] 0x00002b16ff2e2fe7 0.11% perf-80906.map [.] 0x00002b16ff2e2ec0 0.11% [kernel] [k] isolate_migratepages_range 0.09% [kernel] [k] change_pte_range 0.08% perf-80906.map [.] 0x00002b16ff2e2fe3 0.07% [kernel] [k] mem_cgroup_page_lruvec 0.07% perf-80906.map [.] 0x00002b16fcc02086 ``` ---
**Author:** ![bxl](https://avatars.discourse-cdn.com/v4/letter/b/aca169/32.png) [@bxl](https://discuss.elastic.co/u/bxl)\ **Post date:** [March 15, 2021, 8:15am UTC](https://discuss.elastic.co/t/elasticsearch-single-node-is-occasionally-loaded-very-high/266904/8 "2021-03-15T08:15:15Z")
I found that it has something to do with the operating system, in the following environment, bulk write data, the node receiving the write request will appear centos linux release 7.6.1810 3.10.0-957.el7.x86\_64 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 12, 2021, 8:15am UTC](https://discuss.elastic.co/t/elasticsearch-single-node-is-occasionally-loaded-very-high/266904/9 "2021-04-12T08:15:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.