Unassigned Node and shards


(IronMike) #1

I saw this problem twice now.
I start with a Green two-node cluster, default 5 shards/node, I index about
50,000 docs, shards/replicas look great and well balanced across the 2
nodes.

I try the same test with 8 million docs. I come back when its done, and I
see all primary shards on node1 and 2 replicas on node2 and three
unassigned replicas on a third "unassigned" node.

I will look through the logs, but I was wondering if anyone has seen
something similar or has any idea where/why this is coming from before I
dig?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2bddca5e-ea3d-4804-9a0d-47d98bab96c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #2

First, 2 nodes is not a good number for distributed systems, so you must
take extra care that you write to the index with write consistency "full".
Otherwise, your system will be easily exposed to receive split brains, you
get all kind of difficulties, missing documents etc.

Second, check your log files for disconnects and out of memory or other
exceptions. This might give you enough hints about the resources you have
allocated to ES so you can fix the problem.

Third, it would be helpful if you describe your configuration how you index
and what versions you use.

Jörg

On Mon, Aug 18, 2014 at 10:29 PM, IronMan2014 sabdalla80@gmail.com wrote:

I saw this problem twice now.
I start with a Green two-node cluster, default 5 shards/node, I index
about 50,000 docs, shards/replicas look great and well balanced across the
2 nodes.

I try the same test with 8 million docs. I come back when its done, and I
see all primary shards on node1 and 2 replicas on node2 and three
unassigned replicas on a third "unassigned" node.

I will look through the logs, but I was wondering if anyone has seen
something similar or has any idea where/why this is coming from before I
dig?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2bddca5e-ea3d-4804-9a0d-47d98bab96c1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2bddca5e-ea3d-4804-9a0d-47d98bab96c1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHc_f-0hoO%3DgKwrBocBrDJxtxiwWvAKc%2B5O11SYuNxrbw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(IronMike) #3

Thanks. I do realize that I need more nodes, but it was only meant for
testing.
Q1) Can you expand a bit on 'full' writes, I am not sure what you meant by
it (reference also would help.

As far as the setup, its the normal 2 node EC2 cluster, Java transport
Client with Bulk processor.
Q2) My 2 ymls have the default setup for the most part, do I need to tell
one to be a master and the other not? (By default both nodes will be
eligible to be master candidates). As far as the split brain, I thought its
recommended to be 2?N +1, so for 2 nodes, it would be 2.

On Monday, August 18, 2014 5:57:08 PM UTC-4, Jörg Prante wrote:

First, 2 nodes is not a good number for distributed systems, so you must
take extra care that you write to the index with write consistency "full".
Otherwise, your system will be easily exposed to receive split brains, you
get all kind of difficulties, missing documents etc.

Second, check your log files for disconnects and out of memory or other
exceptions. This might give you enough hints about the resources you have
allocated to ES so you can fix the problem.

Third, it would be helpful if you describe your configuration how you
index and what versions you use.

Jörg

On Mon, Aug 18, 2014 at 10:29 PM, IronMan2014 <sabda...@gmail.com
<javascript:>> wrote:

I saw this problem twice now.
I start with a Green two-node cluster, default 5 shards/node, I index
about 50,000 docs, shards/replicas look great and well balanced across the
2 nodes.

I try the same test with 8 million docs. I come back when its done, and I
see all primary shards on node1 and 2 replicas on node2 and three
unassigned replicas on a third "unassigned" node.

I will look through the logs, but I was wondering if anyone has seen
something similar or has any idea where/why this is coming from before I
dig?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2bddca5e-ea3d-4804-9a0d-47d98bab96c1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2bddca5e-ea3d-4804-9a0d-47d98bab96c1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/32658796-0975-4adf-ac5c-e2f6dd023051%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4