# ES master CPU spike suddenly 100% and fails to apply cluster state

**URL:** <https://discuss.elastic.co/t/es-master-cpu-spike-suddenly-100-and-fails-to-apply-cluster-state/251488>\
**Category:** Elasticsearch\
**Created:** [October 8, 2020, 4:55pm UTC](https://discuss.elastic.co/t/es-master-cpu-spike-suddenly-100-and-fails-to-apply-cluster-state/251488 "2020-10-08T16:55:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![UDixit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/udixit/32/53219_2.png) [@UDixit](https://discuss.elastic.co/u/UDixit)\
**Post date:** [October 8, 2020, 4:55pm UTC](https://discuss.elastic.co/t/es-master-cpu-spike-suddenly-100-and-fails-to-apply-cluster-state/251488/1 "2020-10-08T16:55:50Z")

</div>

We are currently operating with ES5.6. All nodes in the ES cluster are master and data eligible.  
The min\_master\_nodes is set to n/2+1.  
We are recently seeing issues that the matser's CPU usage spikes up very high ~90% and all Cluster update task start failing with a timeout.

> TraceLevel="DEBUG" ComponentName="default" Message="[2020-10-08T06:45:53,284][DEBUG][org.elasticsearch.action.admin.cluster.reroute.TransportClusterRerouteAction] failed to perform [cluster\_reroute (api)]  
> ProcessClusterEventTimeoutException[failed to process cluster event (cluster\_reroute (api)) within 30s]  
> at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$null$0(ClusterService.java:255)  
> at java.util.ArrayList.forEach(ArrayList.java:1257)  
> at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.lambda$onTimeout$1(ClusterService.java:254)

Soon after we see multiple logs on the master indicating write and flush on network layer failed

> [org.elasticsearch.transport.netty4.Netty4Transport] write and flush on the network layer failed (channel: [id: 0xb8a365a6, L:0.0.0.0/0.0.0.0:9300 ! R:/xxxxxxxxx])  
> java.nio.channels.ClosedChannelException  
> at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)

We have recently add a new check to our rolling restart logic to continue issuing `allocation/reroute/retry_failed` every 1 minute until the cluster converges.

In rolling restart, since the node is down, the reroute will fail many times before succeeding. Could this be causing extra load on the master?

The cluster remains healthy otherwise but during rolling restart we see such issues happening frequently.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 5, 2020, 4:55pm UTC](https://discuss.elastic.co/t/es-master-cpu-spike-suddenly-100-and-fails-to-apply-cluster-state/251488/2 "2020-11-05T16:55:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.