WARNING: master node with node.data=false deletes cluster

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on a
restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen

This configuration should is valid and should not result in loss of data. I
wonder if you hit something I fixed in master/0.17 branch:
Single node rolling restart into a new node can cause metadata loss · Issue #1249 · elastic/elasticsearch · GitHub.

Can you provide the steps that led to loosing the data? I would like to
recreate it. For example: Start the non_data/master node, then start a
data/non_master node, index some sample data, shut it down, start it in the
same order, ... .

On Tue, Aug 23, 2011 at 6:07 PM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on a
restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen

We are using version 17.6.

The reason to restart the cluster was an update of a plugin.

Here's the scenario:

  • shut down the cluster
  • start the none data node (node.data = false, no other node settings)
    At this time this node was marked as master Node in BigDesk
  • start the data nodes

-> data gone

Unfortunately I can not reproduce this scenario :frowning:

On Tue, Aug 23, 2011 at 8:57 PM, Shay Banon kimchy@gmail.com wrote:

This configuration should is valid and should not result in loss of data. I
wonder if you hit something I fixed in master/0.17 branch:
Single node rolling restart into a new node can cause metadata loss · Issue #1249 · elastic/elasticsearch · GitHub.

Can you provide the steps that led to loosing the data? I would like to
recreate it. For example: Start the non_data/master node, then start a
data/non_master node, index some sample data, shut it down, start it in the
same order, ... .

On Tue, Aug 23, 2011 at 6:07 PM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on a
restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen

I will try and reproduce it as well. I have several automated tests that do
just that, and they pass... . If you manage to get it to happen, would love
to hear it.

On Wed, Aug 24, 2011 at 8:59 AM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

We are using version 17.6.

The reason to restart the cluster was an update of a plugin.

Here's the scenario:

  • shut down the cluster
  • start the none data node (node.data = false, no other node settings)
    At this time this node was marked as master Node in BigDesk
  • start the data nodes

-> data gone

Unfortunately I can not reproduce this scenario :frowning:

On Tue, Aug 23, 2011 at 8:57 PM, Shay Banon kimchy@gmail.com wrote:

This configuration should is valid and should not result in loss of data.
I wonder if you hit something I fixed in master/0.17 branch:
Single node rolling restart into a new node can cause metadata loss · Issue #1249 · elastic/elasticsearch · GitHub.

Can you provide the steps that led to loosing the data? I would like to
recreate it. For example: Start the non_data/master node, then start a
data/non_master node, index some sample data, shut it down, start it in the
same order, ... .

On Tue, Aug 23, 2011 at 6:07 PM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on
a restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen

The cluster consist of 12 data nodes and one none data node.

This is what happend before the cluster restart:

We tried to stop the cluster because all nodes where running on a high load
without a reason.
First we used the _shutdown API without success.
Then we used the shutdown script on all nodes.

On Fri, Aug 26, 2011 at 4:29 PM, Shay Banon kimchy@gmail.com wrote:

I will try and reproduce it as well. I have several automated tests that do
just that, and they pass... . If you manage to get it to happen, would love
to hear it.

On Wed, Aug 24, 2011 at 8:59 AM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

We are using version 17.6.

The reason to restart the cluster was an update of a plugin.

Here's the scenario:

  • shut down the cluster
  • start the none data node (node.data = false, no other node settings)
    At this time this node was marked as master Node in BigDesk
  • start the data nodes

-> data gone

Unfortunately I can not reproduce this scenario :frowning:

On Tue, Aug 23, 2011 at 8:57 PM, Shay Banon kimchy@gmail.com wrote:

This configuration should is valid and should not result in loss of data.
I wonder if you hit something I fixed in master/0.17 branch:
Single node rolling restart into a new node can cause metadata loss · Issue #1249 · elastic/elasticsearch · GitHub.

Can you provide the steps that led to loosing the data? I would like to
recreate it. For example: Start the non_data/master node, then start a
data/non_master node, index some sample data, shut it down, start it in the
same order, ... .

On Tue, Aug 23, 2011 at 6:07 PM, Jürgen kartnaller <
juergen.kartnaller@gmail.com> wrote:

If you have node.data set to false it is important to set node.master to
false.

If a none data node becomes a master the whole cluster data gets lost on
a restart!

@Shay
It would be a good idea to automatically set node.master to false if
node.data is set to true.

Jürgen