# One of the master pod is going to crashloop backoff

**URL:** <https://discuss.elastic.co/t/one-of-the-master-pod-is-going-to-crashloop-backoff/265977>\
**Category:** Elasticsearch\
**Created:** [March 2, 2021, 3:28pm UTC](https://discuss.elastic.co/t/one-of-the-master-pod-is-going-to-crashloop-backoff/265977 "2021-03-02T15:28:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![knagasri](https://avatars.discourse-cdn.com/v4/letter/k/47e85d/32.png) [@knagasri](https://discuss.elastic.co/u/knagasri)\
**Post date:** [March 2, 2021, 3:28pm UTC](https://discuss.elastic.co/t/one-of-the-master-pod-is-going-to-crashloop-backoff/265977/1 "2021-03-02T15:28:55Z")

</div>

We have deployed Elasticsearch cluster with 7.4.0 and operator version 1.1.0 in GKE .One of the master pod is going to crashloop backoff without doing any changes. I can see below error when I checked the logs

```auto
"org.elasticsearch.ElasticsearchException: node [{elasticsearch-cluster-es-master-2}{g2EpPknaRwCYZv4sGx1yTw}{YaFEhY43SzaO760YLTo4Mw}{240.0.141.139}{240.0.141.139:9300}{lm}{ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true}] failed [3] consecutive checks"

```

I was following some vlogs already created.([Unstable after starting nodes - #15 by coudenysj](https://discuss.elastic.co/t/unstable-after-starting-nodes/200526/15)). But didn't find exact solution. Can anyone tell the reason why it is failing

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 2, 2021, 7:35pm UTC](https://discuss.elastic.co/t/one-of-the-master-pod-is-going-to-crashloop-backoff/265977/2 "2021-03-02T19:35:02Z")

</div>

Is there more to the log?

---

<div class="post-metadata">

**Author:** ![knagasri](https://avatars.discourse-cdn.com/v4/letter/k/47e85d/32.png) [@knagasri](https://discuss.elastic.co/u/knagasri)\
**Post date:** [March 3, 2021, 5:27am UTC](https://discuss.elastic.co/t/one-of-the-master-pod-is-going-to-crashloop-backoff/265977/3 "2021-03-03T05:27:54Z")

</div>

providing few more logs.

```auto
"stacktrace": ["org.elasticsearch.ElasticsearchException: node [{elasticsearch-cluster-es-master-2}{g2EpPknaRwCYZv4sGx1yTw}{YaFEhY43SzaO760YLTo4Mw}{240.0.141.139}{240.0.141.139:9300}{lm}{ml.machine_memory=17179869184, ml.max_open_jobs=20, xpack.installed=true}] failed [3] consecutive checks",
"at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:278) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:243) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:241) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:233) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:136) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:663) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]",
"at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[?:?]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) ~[?:?]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597) ~[?:?]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551) ~[?:?]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) ~[?:?]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[?:?]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]",
"Caused by: org.elasticsearch.transport.RemoteTransportException: [elasticsearch-cluster-es-master-2][internal:coordination/fault_detection/leader_check]",
"Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: leader check from unknown node"
"at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:181) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:105) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:376) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:184) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:246) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) ~[?:?]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:185) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:118) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:663) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]",
"at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) ~[?:?]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[?:?]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) ~[?:?]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597) ~[?:?]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551) ~[?:?]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) ~[?:?]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[?:?]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]",
"at java.lang.Thread.run(Thread.java:830) ~[?:?]"] }

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 3, 2021, 6:08am UTC](https://discuss.elastic.co/t/one-of-the-master-pod-is-going-to-crashloop-backoff/265977/4 "2021-03-03T06:08:35Z")

</div>

Please provide a larger section of logs that provide context of what happened before and after the sections you posted.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 31, 2021, 6:09am UTC](https://discuss.elastic.co/t/one-of-the-master-pod-is-going-to-crashloop-backoff/265977/5 "2021-03-31T06:09:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.