Accidently ran another instance of elasticsearch on a few nodes

Elastic Stack Elasticsearch
1

Hi, while trying to set up an another process as master, i believe i for
some time ran multiple instances of elasticsearch on three nodes.

On these nodes, its looks like this:
/var/lib/elasticsearch/elasticsearch/indices/0
/var/lib/elasticsearch/elasticsearch/indices/1

On my other nodes, that are fine it looks like:
/var/lib/elasticsearch/elasticsearch/indices/0

So, there is alot of data in the "1"-directory on three nodes, and these
shards will not be ASSIGNED, my cluster stays yellow.

This mistake happend 1 week ago, since then i have restarted ES a couple of
times, but it was just now that i got the problem.

How can i fix this?

At the moment im running 5 nodes, where 3 runs another instance of
elasticsearch, as just masters

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3c852fff-611a-45fb-b1c2-e5962d733977%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Yellow means unassigned replicas, try removing them and then adding them
back.

Once your cluster is green you can stop one of the nodes with the extra
data and then delete the extra directory, just make sure you let the other
nodes rebalance and your cluster is green again before deleting, otherwise
you may risk losing data.

On 8 November 2014 19:12, Johan Öhr johan.ohr@gmail.com wrote:

Hi, while trying to set up an another process as master, i believe i for
some time ran multiple instances of elasticsearch on three nodes.

On these nodes, its looks like this:
/var/lib/elasticsearch/elasticsearch/indices/0
/var/lib/elasticsearch/elasticsearch/indices/1

On my other nodes, that are fine it looks like:
/var/lib/elasticsearch/elasticsearch/indices/0

So, there is alot of data in the "1"-directory on three nodes, and these
shards will not be ASSIGNED, my cluster stays yellow.

This mistake happend 1 week ago, since then i have restarted ES a couple
of times, but it was just now that i got the problem.

How can i fix this?

At the moment im running 5 nodes, where 3 runs another instance of
elasticsearch, as just masters

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3c852fff-611a-45fb-b1c2-e5962d733977%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3c852fff-611a-45fb-b1c2-e5962d733977%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZnUn1hHAy60TTiOxo8b5rdt0N35dDYjq7Abb3M8pNNEjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

To avoid this situation in the future, besides using a service to start
elasticsearch, you can enforce the max nodes setting:

node.max_local_storage_nodes: 1

--
Ivan

On Sun, Nov 9, 2014 at 5:24 PM, Mark Walkom markwalkom@gmail.com wrote:

Yellow means unassigned replicas, try removing them and then adding them
back.

Once your cluster is green you can stop one of the nodes with the extra
data and then delete the extra directory, just make sure you let the other
nodes rebalance and your cluster is green again before deleting, otherwise
you may risk losing data.

On 8 November 2014 19:12, Johan Öhr johan.ohr@gmail.com wrote:

Hi, while trying to set up an another process as master, i believe i for
some time ran multiple instances of elasticsearch on three nodes.

On these nodes, its looks like this:
/var/lib/elasticsearch/elasticsearch/indices/0
/var/lib/elasticsearch/elasticsearch/indices/1

On my other nodes, that are fine it looks like:
/var/lib/elasticsearch/elasticsearch/indices/0

So, there is alot of data in the "1"-directory on three nodes, and these
shards will not be ASSIGNED, my cluster stays yellow.

This mistake happend 1 week ago, since then i have restarted ES a couple
of times, but it was just now that i got the problem.

How can i fix this?

At the moment im running 5 nodes, where 3 runs another instance of
elasticsearch, as just masters

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3c852fff-611a-45fb-b1c2-e5962d733977%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3c852fff-611a-45fb-b1c2-e5962d733977%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZnUn1hHAy60TTiOxo8b5rdt0N35dDYjq7Abb3M8pNNEjg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZnUn1hHAy60TTiOxo8b5rdt0N35dDYjq7Abb3M8pNNEjg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBUueXaZ64g5%2B%2B3R69iFxJWo3oHQXq3cuhKt4CfYB33Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Thank you

I fixed the problem by taking down a node, delete wrong directory, start the node. It worked fine with the first two nodes. But not the third ..

See https://groups.google.com/forum/m/?utm_medium=email&utm_source=footer#!topic/elasticsearch/7LZVzQkcAtA

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dd0258ee-bc4d-4b05-b3f8-98985e71da62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.