What is the proper way to restart node without clients noticing?

Hi

I have a cluster of few es nodes. Due to changes in java scripts, or
maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to
false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going to shut down
this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

--
Maciej Dziardziel (fiedzia@gmail.com)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

If your replication values and quorum settings are correct, you can disable
allocation and simply restart the nodes. Assuming the shards are correctly
distributed, the other nodes will contain all the search data. It helps to
disable indexing as well and flush the transaction logs so that you can
return the the original state after the nodes rejoin and allocation is
reenabled.

Cheers,

Ivan

On Tue, Jun 18, 2013 at 6:06 AM, Maciej Dziardziel fiedzia@gmail.comwrote:

Hi

I have a cluster of few es nodes. Due to changes in java scripts, or
maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to
false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going to shut down
this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

--
Maciej Dziardziel (fiedzia@gmail.com)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi

Thanks for your answer. Yes, other nodes will contain data and new requests
can be handled by them. The problem is that:

  1. Every request from a client connected to the node i am going to restart
    will fail when i do that.
  2. It will take time and some failed requests for a client to realize that
    a node is down and it should speak to others.

That's the problems i was trying to avoid.

Since i can't find anything useful in ES documentation, i worked around it
with a load balancer that allows me to gracefully remove nodes
(without breaking open connections). It would be handy to have such feature
in ES by default though.

On Wednesday, June 19, 2013 3:14:00 PM UTC+1, Ivan Brusic wrote:

If your replication values and quorum settings are correct, you can
disable allocation and simply restart the nodes. Assuming the shards are
correctly distributed, the other nodes will contain all the search data. It
helps to disable indexing as well and flush the transaction logs so that
you can return the the original state after the nodes rejoin and allocation
is reenabled.

Cheers,

Ivan

On Tue, Jun 18, 2013 at 6:06 AM, Maciej Dziardziel <fie...@gmail.com<javascript:>

wrote:

Hi

I have a cluster of few es nodes. Due to changes in java scripts, or
maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to
false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going to shut down
this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

--
Maciej Dziardziel (fie...@gmail.com <javascript:>)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Are you using the TransportClient or the REST client? The TransportClient
should handle most of these issues. Can't really help if you are using HTTP.

Cheers,

Ivan

On Wed, Jun 19, 2013 at 3:08 PM, Maciej Dziardziel fiedzia@gmail.comwrote:

Hi

Thanks for your answer. Yes, other nodes will contain data and new
requests can be handled by them. The problem is that:

  1. Every request from a client connected to the node i am going to restart
    will fail when i do that.
  2. It will take time and some failed requests for a client to realize that
    a node is down and it should speak to others.

That's the problems i was trying to avoid.

Since i can't find anything useful in ES documentation, i worked around it
with a load balancer that allows me to gracefully remove nodes
(without breaking open connections). It would be handy to have such
feature in ES by default though.

On Wednesday, June 19, 2013 3:14:00 PM UTC+1, Ivan Brusic wrote:

If your replication values and quorum settings are correct, you can
disable allocation and simply restart the nodes. Assuming the shards are
correctly distributed, the other nodes will contain all the search data. It
helps to disable indexing as well and flush the transaction logs so that
you can return the the original state after the nodes rejoin and allocation
is reenabled.

Cheers,

Ivan

On Tue, Jun 18, 2013 at 6:06 AM, Maciej Dziardziel fie...@gmail.comwrote:

Hi

I have a cluster of few es nodes. Due to changes in java scripts, or
maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to
false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going to shut
down this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

--
Maciej Dziardziel (fie...@gmail.com)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

Once used GitHub - exalead/commons-httpclient-failover: Load-balancing and failover support for Apache HttpClient 3.X with
elasticsearch multiple nodes in a cluster.
That was long back though, not sure whether its still relevant or not.

  • Sujoy.

On Thursday, June 20, 2013 3:38:02 AM UTC+5:30, Maciej Dziardziel wrote:

Hi

Thanks for your answer. Yes, other nodes will contain data and new
requests can be handled by them. The problem is that:

  1. Every request from a client connected to the node i am going to restart
    will fail when i do that.
  2. It will take time and some failed requests for a client to realize that
    a node is down and it should speak to others.

That's the problems i was trying to avoid.

Since i can't find anything useful in ES documentation, i worked around it
with a load balancer that allows me to gracefully remove nodes
(without breaking open connections). It would be handy to have such
feature in ES by default though.

On Wednesday, June 19, 2013 3:14:00 PM UTC+1, Ivan Brusic wrote:

If your replication values and quorum settings are correct, you can
disable allocation and simply restart the nodes. Assuming the shards are
correctly distributed, the other nodes will contain all the search data. It
helps to disable indexing as well and flush the transaction logs so that
you can return the the original state after the nodes rejoin and allocation
is reenabled.

Cheers,

Ivan

On Tue, Jun 18, 2013 at 6:06 AM, Maciej Dziardziel fie...@gmail.comwrote:

Hi

I have a cluster of few es nodes. Due to changes in java scripts, or
maintenance, sometimes i need to restart them.
What is the proper way of doing it without clients noticing?

I was thinking about setting up additional nodes with node.data set to
false. Clients would connect to those non-data nodes.
Now i would only need a way to say to es cluster "i'm going to shut
down this node in 10 minutes, so let it finish processing its requests,
and redirect new ones to other nodes". Is it possibe?

--
Maciej Dziardziel (fie...@gmail.com)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.