WARNING: master node with node.data=false deletes cluster


(Jürgen kartnaller) #1

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on a
restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen


(Shay Banon) #2

This configuration should is valid and should not result in loss of data. I
wonder if you hit something I fixed in master/0.17 branch:
https://github.com/elasticsearch/elasticsearch/issues/1249.

Can you provide the steps that led to loosing the data? I would like to
recreate it. For example: Start the non_data/master node, then start a
data/non_master node, index some sample data, shut it down, start it in the
same order, ... .

On Tue, Aug 23, 2011 at 6:07 PM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on a
restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen


(Jürgen kartnaller) #3

We are using version 17.6.

The reason to restart the cluster was an update of a plugin.

Here's the scenario:

  • shut down the cluster
  • start the none data node (node.data = false, no other node settings)
    At this time this node was marked as master Node in BigDesk
  • start the data nodes

-> data gone

Unfortunately I can not reproduce this scenario :frowning:

On Tue, Aug 23, 2011 at 8:57 PM, Shay Banon kimchy@gmail.com wrote:

This configuration should is valid and should not result in loss of data. I
wonder if you hit something I fixed in master/0.17 branch:
https://github.com/elasticsearch/elasticsearch/issues/1249.

Can you provide the steps that led to loosing the data? I would like to
recreate it. For example: Start the non_data/master node, then start a
data/non_master node, index some sample data, shut it down, start it in the
same order, ... .

On Tue, Aug 23, 2011 at 6:07 PM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on a
restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen


(Shay Banon) #4

I will try and reproduce it as well. I have several automated tests that do
just that, and they pass... . If you manage to get it to happen, would love
to hear it.

On Wed, Aug 24, 2011 at 8:59 AM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

We are using version 17.6.

The reason to restart the cluster was an update of a plugin.

Here's the scenario:

  • shut down the cluster
  • start the none data node (node.data = false, no other node settings)
    At this time this node was marked as master Node in BigDesk
  • start the data nodes

-> data gone

Unfortunately I can not reproduce this scenario :frowning:

On Tue, Aug 23, 2011 at 8:57 PM, Shay Banon kimchy@gmail.com wrote:

This configuration should is valid and should not result in loss of data.
I wonder if you hit something I fixed in master/0.17 branch:
https://github.com/elasticsearch/elasticsearch/issues/1249.

Can you provide the steps that led to loosing the data? I would like to
recreate it. For example: Start the non_data/master node, then start a
data/non_master node, index some sample data, shut it down, start it in the
same order, ... .

On Tue, Aug 23, 2011 at 6:07 PM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on
a restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen


(Jürgen kartnaller) #5

The cluster consist of 12 data nodes and one none data node.

This is what happend before the cluster restart:

We tried to stop the cluster because all nodes where running on a high load
without a reason.
First we used the _shutdown API without success.
Then we used the shutdown script on all nodes.

On Fri, Aug 26, 2011 at 4:29 PM, Shay Banon kimchy@gmail.com wrote:

I will try and reproduce it as well. I have several automated tests that do
just that, and they pass... . If you manage to get it to happen, would love
to hear it.

On Wed, Aug 24, 2011 at 8:59 AM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

We are using version 17.6.

The reason to restart the cluster was an update of a plugin.

Here's the scenario:

  • shut down the cluster
  • start the none data node (node.data = false, no other node settings)
    At this time this node was marked as master Node in BigDesk
  • start the data nodes

-> data gone

Unfortunately I can not reproduce this scenario :frowning:

On Tue, Aug 23, 2011 at 8:57 PM, Shay Banon kimchy@gmail.com wrote:

This configuration should is valid and should not result in loss of data.
I wonder if you hit something I fixed in master/0.17 branch:
https://github.com/elasticsearch/elasticsearch/issues/1249.

Can you provide the steps that led to loosing the data? I would like to
recreate it. For example: Start the non_data/master node, then start a
data/non_master node, index some sample data, shut it down, start it in the
same order, ... .

On Tue, Aug 23, 2011 at 6:07 PM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on
a restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen


(system) #6