# Errors during node restart
**URL:** https://discuss.elastic.co/t/errors-during-node-restart/14710
**Category:** Elasticsearch
**Created:** [December 4, 2013, 5:14pm UTC](https://discuss.elastic.co/t/errors-during-node-restart/14710 "2013-12-04T17:14:05Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@slushi](https://discuss.elastic.co/u/slushi)
#### Post date: [December 4, 2013, 5:14pm UTC](https://discuss.elastic.co/t/errors-during-node-restart/14710/1 "2013-12-04T17:14:05Z")
We have a 2 node elasticsearch cluster serving an index with 1 shard and
replica. Whenever we restart a node, we get a handful of exceptions. We use
the transport client, looking at the code it seems like it just attempts
the request on each node before finally failing, so we are not sure why we
are seeing failures. Perhaps the search request is accepted by the node but
the node is not ready to process it?
The stack is below:
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked
by: [SERVICE\_UNAVAILABLE/1/state not recovered /
initialized];[SERVICE\_UNAVAILABLE/2/no master];
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
at
org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.(TransportSearchTypeAction.java:107)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.(TransportSearchQueryThenFetchAction.java:68)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.(TransportSearchQueryThenFetchAction.java:62)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:59)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:49)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:108)
at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at
org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:135)
at
org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:120)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:212)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:109)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cb32dd4a-2aa6-4800-9f9e-8edd8a9988ec%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cb32dd4a-2aa6-4800-9f9e-8edd8a9988ec%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [December 5, 2013, 9:56am UTC](https://discuss.elastic.co/t/errors-during-node-restart/14710/2 "2013-12-05T09:56:03Z")
Hey,
do you startup the node and connect immediately to the node without waiting
for it to join the cluster? The node needs to join the cluster first, and
your exception basically shows that the node has not found a master yet.
--Alex
On Wed, Dec 4, 2013 at 6:14 PM, slushi [kireetreddy@gmail.com](mailto:kireetreddy@gmail.com) wrote:
> We have a 2 node elasticsearch cluster serving an index with 1 shard and
> replica. Whenever we restart a node, we get a handful of exceptions. We use
> the transport client, looking at the code it seems like it just attempts
> the request on each node before finally failing, so we are not sure why we
> are seeing failures. Perhaps the search request is accepted by the node but
> the node is not ready to process it?
>
> The stack is below:
>
> Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked
> by: [SERVICE\_UNAVAILABLE/1/state not recovered /
> initialized];[SERVICE\_UNAVAILABLE/2/no master];
> at
> org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
> at
> org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
> at
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.(TransportSearchTypeAction.java:107)
> at
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.(TransportSearchQueryThenFetchAction.java:68)
> at
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.(TransportSearchQueryThenFetchAction.java:62)
> at
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:59)
> at
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:49)
> at
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
> at
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:108)
> at
> org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
> at
> org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
> at
> org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:135)
> at
> org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:120)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:212)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:109)
> at
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> at
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
> at
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> at
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> ... 3 more
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/cb32dd4a-2aa6-4800-9f9e-8edd8a9988ec%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cb32dd4a-2aa6-4800-9f9e-8edd8a9988ec%40googlegroups.com)
> .
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_RNXCqPt15Bj10%3DKp0SJGT-u7j%2B0PcqA\_E1m-xNQJWKg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_RNXCqPt15Bj10%3DKp0SJGT-u7j%2B0PcqA_E1m-xNQJWKg%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@slushi](https://discuss.elastic.co/u/slushi)
#### Post date: [December 5, 2013, 3:13pm UTC](https://discuss.elastic.co/t/errors-during-node-restart/14710/3 "2013-12-05T15:13:53Z")
Well, what we are trying to do is keep our application up while restarting
a node. We may do this if we need to service the machine or do a rolling
upgrade of the cluster with no application downtime. So I suppose we aren't
waiting for the node to join the cluster -- we assumed that the transport
client would just take care of this (at least on a graceful node restart).
Shouldn't the elasticsearch client code handle this? If not, is the
workaround to catch this particular exception and do a retry in our
application code? If a node crashes, would we need to do something similar?
On Thursday, December 5, 2013 4:56:03 AM UTC-5, Alexander Reelsen wrote:
> Hey,
>
> do you startup the node and connect immediately to the node without
> waiting for it to join the cluster? The node needs to join the cluster
> first, and your exception basically shows that the node has not found a
> master yet.
>
> --Alex
>
> On Wed, Dec 4, 2013 at 6:14 PM, slushi \<[kiree...@gmail.com](mailto:kiree...@gmail.com) \\>wrote:
>
> > We have a 2 node elasticsearch cluster serving an index with 1 shard and
> > replica. Whenever we restart a node, we get a handful of exceptions. We use
> > the transport client, looking at the code it seems like it just attempts
> > the request on each node before finally failing, so we are not sure why we
> > are seeing failures. Perhaps the search request is accepted by the node but
> > the node is not ready to process it?
> >
> > The stack is below:
> >
> > Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked
> > by: [SERVICE\_UNAVAILABLE/1/state not recovered /
> > initialized];[SERVICE\_UNAVAILABLE/2/no master];
> > at
> > org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
> > at
> > org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:128)
> > at
> > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.(TransportSearchTypeAction.java:107)
> > at
> > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.(TransportSearchQueryThenFetchAction.java:68)
> > at
> > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.(TransportSearchQueryThenFetchAction.java:62)
> > at
> > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:59)
> > at
> > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:49)
> > at
> > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
> > at
> > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:108)
> > at
> > org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
> > at
> > org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
> > at
> > org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:135)
> > at
> > org.elasticsearch.action.search.TransportSearchAction$TransportHandler.messageReceived(TransportSearchAction.java:120)
> > at
> > org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:212)
> > at
> > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:109)
> > at
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> > at
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > at
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> > at
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > at
> > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> > at
> > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
> > at
> > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> > at
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> > at
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > at
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> > at
> > org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
> > at
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > at
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > at
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > at
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > at
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> > at
> > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> > at
> > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> > at
> > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> > at
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > at
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > at
> > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > ... 3 more
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/cb32dd4a-2aa6-4800-9f9e-8edd8a9988ec%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cb32dd4a-2aa6-4800-9f9e-8edd8a9988ec%40googlegroups.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/291529de-5a17-4d20-86e6-75402fc2c7a7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/291529de-5a17-4d20-86e6-75402fc2c7a7%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:03am UTC](https://discuss.elastic.co/t/errors-during-node-restart/14710/4 "2017-07-06T02:03:05Z")