Unassigned shards

Elastic Stack Elasticsearch
1

Hi,
i am using elasticsearch for a while and one morning i have noticed i have
an unassigned shard.
i started to play with the cluster (restart some nodes, add more nodes, try
to move allocate the shards) and got some more unassigned
i realized all the shards which are primary in one of the nodes has their
replicas unassigned.
also, i tired to allocate or move those shards, and all i get is:

org.elasticsearch.transport.RemoteTransportException:
[foa02][inet[/192.168.2.2:9300]][cluster/reroute]
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException:
[allocate] allocation of [logstash-2014.02.18][0] on node
[foa8][Rhmt4uF6Rf2_8WAw-WieZA][inet[192.168.2.8/192.168.2.8:9300]]{master=false}
is not allowed, reason: NO()
at
org.elasticsearch.cluster.routing.allocation.command.AllocateAllocationCommand.execute(AllocateAllocationCommand.java:188)
at
org.elasticsearch.cluster.routing.allocation.command.AllocationCommands.execute(AllocationCommands.java:116)
at
org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:125)
at
org.elasticsearch.action.admin.cluster.reroute.TransportClusterRerouteAction$1.execute(TransportClusterRerouteAction.java:112)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:300)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
i attached an image from the "head" plugin

https://lh4.googleusercontent.com/-lQxXqu8G1jc/UwSh5GyXkuI/AAAAAAAAGV0/ZfnsveGeA30/s1600/head.jpg

any idea what is wrong?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c2a5ba79-3bfd-41c5-981f-6e535c6c0052%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

There's a lot of info missing: how many servers, indexes, shards and
replicas do you have? have you made sure to set all the configurations
correctly (quorum size, expected number of nodes etc)?

You may be experiencing a split-brain situation (if your cluster state is
red) or just a temporary hiccup (if it's yellow). But without more info its
hard to tell. Either way, restarting the entire cluster usually helps
immediately, if this is an option.

With regards to the response "is not allowed, reason: NO()" you are
getting, there is an open issue on this for getting more feedback in the
response, that I think was made it to v1 and maybe even 0.90.11.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Wed, Feb 19, 2014 at 2:22 PM, Tal Shemesh tal.shemesh@gmail.com wrote:

Hi,
i am using elasticsearch for a while and one morning i have noticed i have
an unassigned shard.
i started to play with the cluster (restart some nodes, add more nodes,
try to move allocate the shards) and got some more unassigned
i realized all the shards which are primary in one of the nodes has their
replicas unassigned.
also, i tired to allocate or move those shards, and all i get is:

org.elasticsearch.transport.RemoteTransportException:
[foa02][inet[/192.168.2.2:9300]][cluster/reroute]
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException:
[allocate] allocation of [logstash-2014.02.18][0] on node
[foa8][Rhmt4uF6Rf2_8WAw-WieZA][inet[
192.168.2.8/192.168.2.8:9300]]{master=false}http://192.168.2.8/192.168.2.8:9300]]{master=false}is not allowed, reason: NO()
at
org.elasticsearch.cluster.routing.allocation.command.AllocateAllocationCommand.execute(AllocateAllocationCommand.java:188)
at
org.elasticsearch.cluster.routing.allocation.command.AllocationCommands.execute(AllocationCommands.java:116)
at
org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:125)
at
org.elasticsearch.action.admin.cluster.reroute.TransportClusterRerouteAction$1.execute(TransportClusterRerouteAction.java:112)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:300)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
i attached an image from the "head" plugin

https://lh4.googleusercontent.com/-lQxXqu8G1jc/UwSh5GyXkuI/AAAAAAAAGV0/ZfnsveGeA30/s1600/head.jpg

any idea what is wrong?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c2a5ba79-3bfd-41c5-981f-6e535c6c0052%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Ztk7J1tQMnxCD851L--i6%2BL20vVU47svmu%2B11d9Y_a5sw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

ok, i found the problem.

the problematic server was in the wrong version of elasticsearch.
i hope it will help others :slight_smile:

thanks.

On Wed, Feb 19, 2014 at 2:29 PM, Itamar Syn-Hershko itamar@code972.comwrote:

There's a lot of info missing: how many servers, indexes, shards and
replicas do you have? have you made sure to set all the configurations
correctly (quorum size, expected number of nodes etc)?

You may be experiencing a split-brain situation (if your cluster state is
red) or just a temporary hiccup (if it's yellow). But without more info its
hard to tell. Either way, restarting the entire cluster usually helps
immediately, if this is an option.

With regards to the response "is not allowed, reason: NO()" you are
getting, there is an open issue on this for getting more feedback in the
response, that I think was made it to v1 and maybe even 0.90.11.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Wed, Feb 19, 2014 at 2:22 PM, Tal Shemesh tal.shemesh@gmail.comwrote:

Hi,
i am using elasticsearch for a while and one morning i have noticed i
have an unassigned shard.
i started to play with the cluster (restart some nodes, add more nodes,
try to move allocate the shards) and got some more unassigned
i realized all the shards which are primary in one of the nodes has their
replicas unassigned.
also, i tired to allocate or move those shards, and all i get is:

org.elasticsearch.transport.RemoteTransportException:
[foa02][inet[/192.168.2.2:9300]][cluster/reroute]
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException:
[allocate] allocation of [logstash-2014.02.18][0] on node
[foa8][Rhmt4uF6Rf2_8WAw-WieZA][inet[
192.168.2.8/192.168.2.8:9300]]{master=false}http://192.168.2.8/192.168.2.8:9300]]{master=false}is not allowed, reason: NO()
at
org.elasticsearch.cluster.routing.allocation.command.AllocateAllocationCommand.execute(AllocateAllocationCommand.java:188)
at
org.elasticsearch.cluster.routing.allocation.command.AllocationCommands.execute(AllocationCommands.java:116)
at
org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:125)
at
org.elasticsearch.action.admin.cluster.reroute.TransportClusterRerouteAction$1.execute(TransportClusterRerouteAction.java:112)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:300)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
i attached an image from the "head" plugin

https://lh4.googleusercontent.com/-lQxXqu8G1jc/UwSh5GyXkuI/AAAAAAAAGV0/ZfnsveGeA30/s1600/head.jpg

any idea what is wrong?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c2a5ba79-3bfd-41c5-981f-6e535c6c0052%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/1aVNnWNnIPs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4Ztk7J1tQMnxCD851L--i6%2BL20vVU47svmu%2B11d9Y_a5sw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAfe%2B1GLm88yKCx8KS%3DwkMPQzTvCPrdr-ALMDX_1oNaLRJtQ8Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.