ES cluster unable to assing new shards

Elastic Stack Elasticsearch
1

Hello there,

on Monday morning our ES cluster cluster switched to red, because he didn't
assign the new created indices to any ES node, see attached picture.

I tried manually allocating these unassigned shards to a node, but it only
returned the following error:

curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node": "es@log09", "allow_primary": true }}]}'

{
"error" : "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]]; nested: IllegalFormatConversionException[d != java.lang.Double]; ",
"status" : 400
}

Also see https://gist.github.com/xorpaul/10644099

I also tried

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.disable_allocation": false}'

and

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.enable": "all"}'

and removing one node from the cluster, which seemed to help, because it
finally assigned the shards to a node.

Unfortunately the same problem appeared again the next day, when the
cluster tried to create new indices.

Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy

I would like to find out the reason why the cluster doesn't assign these
new shards to any node or find a way to issue a command to the cluster to
reassign/redistribute all unassigned shards to a node.

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Check your ES logs, there may be something there.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 15 April 2014 22:20, Andreas Paul xorpaul@gmail.com wrote:

Hello there,

on Monday morning our ES cluster cluster switched to red, because he
didn't assign the new created indices to any ES node, see attached picture.

I tried manually allocating these unassigned shards to a node, but it only
returned the following error:

curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node": "es@log09", "allow_primary": true }}]}'

{
"error" : "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]]; nested: IllegalFormatConversionException[d != java.lang.Double]; ",
"status" : 400
}

Also see Elasticsearch Cluster · GitHub

I also tried

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.disable_allocation": false}'

and

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.enable": "all"}'

and removing one node from the cluster, which seemed to help, because it
finally assigned the shards to a node.

Unfortunately the same problem appeared again the next day, when the
cluster tried to create new indices.

Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy

I would like to find out the reason why the cluster doesn't assign these
new shards to any node or find a way to issue a command to the cluster to
reassign/redistribute all unassigned shards to a node.

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bko7rK%3Ds%2Bp9oJFdCW9VdykCusUP0DJER8%3Dj%3Dj9%2BtLLBQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Mark,

I forgot to write it again in this mail, but in the gist I pasted the full
logs when the ES cluster created the new indices until I tried to restart
the current active master.

head es_cluster.log

[2014-04-14 02:00:01,504][INFO ][cluster.metadata ] [es@log01] [logstash-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1], mappings [default]
[2014-04-14 02:00:02,938][INFO ][cluster.metadata ] [es@log01] [puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1], mappings
[2014-04-14 10:46:12,318][INFO ][node ] [es@log01] stopping ...
[2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)

Which means that there are no log messages at all on why the cluster
couldn't assign the new shards.

On Tuesday, April 15, 2014 5:07:12 PM UTC+2, Mark Walkom wrote:

Check your ES logs, there may be something there.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 15 April 2014 22:20, Andreas Paul <xor...@gmail.com <javascript:>>wrote:

Hello there,

on Monday morning our ES cluster cluster switched to red, because he
didn't assign the new created indices to any ES node, see attached picture.

I tried manually allocating these unassigned shards to a node, but it
only returned the following error:

curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node": "es@log09", "allow_primary": true }}]}'

{
"error" : "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]]; nested: IllegalFormatConversionException[d != java.lang.Double]; ",
"status" : 400

}

Also see Elasticsearch Cluster · GitHub

I also tried

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.disable_allocation": false}'

and

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.enable": "all"}'

and removing one node from the cluster, which seemed to help, because it
finally assigned the shards to a node.

Unfortunately the same problem appeared again the next day, when the
cluster tried to create new indices.

Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy

I would like to find out the reason why the cluster doesn't assign these
new shards to any node or find a way to issue a command to the cluster to
reassign/redistribute all unassigned shards to a node.

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2cb5fb25-8687-47a0-bb98-40b7a01074c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Yesterday I set the replica count to 0 with

curl -XPUT $(hostname -f):9200/_settings -d'{'index': {
'number_of_replicas':0}}'

and today the ES cluster assigned the new shards as it should.

I have now set the replica count back to 1 and will see if that's the
problem tomorrow.

On Tuesday, April 15, 2014 5:43:32 PM UTC+2, Andreas Paul wrote:

Hi Mark,

I forgot to write it again in this mail, but in the gist I pasted the full
logs when the ES cluster created the new indices until I tried to restart
the current active master.

head es_cluster.log

[2014-04-14 02:00:01,504][INFO ][cluster.metadata ] [es@log01] [logstash-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1], mappings [default]
[2014-04-14 02:00:02,938][INFO ][cluster.metadata ] [es@log01] [puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1], mappings
[2014-04-14 10:46:12,318][INFO ][node ] [es@log01] stopping ...
[2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)

Which means that there are no log messages at all on why the cluster
couldn't assign the new shards.

On Tuesday, April 15, 2014 5:07:12 PM UTC+2, Mark Walkom wrote:

Check your ES logs, there may be something there.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 15 April 2014 22:20, Andreas Paul xor...@gmail.com wrote:

Hello there,

on Monday morning our ES cluster cluster switched to red, because he
didn't assign the new created indices to any ES node, see attached picture.

I tried manually allocating these unassigned shards to a node, but it
only returned the following error:

curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node": "es@log09", "allow_primary": true }}]}'

{
"error" : "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]]; nested: IllegalFormatConversionException[d != java.lang.Double]; ",
"status" : 400

}

Also see Elasticsearch Cluster · GitHub

I also tried

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.disable_allocation": false}'

and

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.enable": "all"}'

and removing one node from the cluster, which seemed to help, because it
finally assigned the shards to a node.

Unfortunately the same problem appeared again the next day, when the
cluster tried to create new indices.

Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy

I would like to find out the reason why the cluster doesn't assign these
new shards to any node or find a way to issue a command to the cluster to
reassign/redistribute all unassigned shards to a node.

Thanks in advance!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9c825b3c-5556-46fd-9ed2-4c86df44f7ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Today the ES cluster still works as expected.

Still don't know the reason why it failed in the first place or what I did
to fix it.
Maybe a slow cluster restart helped: stopping all nodes and then starting
only one node so it can become master instead of restarting all at once and
letting them work it out who should become master.

Maybe I had a split brain problem by restarting too quickly, but then why
would I see all nodes in the cluster information with only one master.

Anyway it's working now...

On Wednesday, April 16, 2014 10:26:15 AM UTC+2, Andreas Paul wrote:

Yesterday I set the replica count to 0 with

curl -XPUT $(hostname -f):9200/_settings -d'{'index': {
'number_of_replicas':0}}'

and today the ES cluster assigned the new shards as it should.

I have now set the replica count back to 1 and will see if that's the
problem tomorrow.

On Tuesday, April 15, 2014 5:43:32 PM UTC+2, Andreas Paul wrote:

Hi Mark,

I forgot to write it again in this mail, but in the gist I pasted the
full logs when the ES cluster created the new indices until I tried to
restart the current active master.

head es_cluster.log

[2014-04-14 02:00:01,504][INFO ][cluster.metadata ] [es@log01] [logstash-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1], mappings [default]
[2014-04-14 02:00:02,938][INFO ][cluster.metadata ] [es@log01] [puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1], mappings
[2014-04-14 10:46:12,318][INFO ][node ] [es@log01] stopping ...
[2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)

Which means that there are no log messages at all on why the cluster
couldn't assign the new shards.

On Tuesday, April 15, 2014 5:07:12 PM UTC+2, Mark Walkom wrote:

Check your ES logs, there may be something there.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 15 April 2014 22:20, Andreas Paul xor...@gmail.com wrote:

Hello there,

on Monday morning our ES cluster cluster switched to red, because he
didn't assign the new created indices to any ES node, see attached picture.

I tried manually allocating these unassigned shards to a node, but it
only returned the following error:

curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node": "es@log09", "allow_primary": true }}]}'

{
"error" : "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]]; nested: IllegalFormatConversionException[d != java.lang.Double]; ",
"status" : 400

}

Also see Elasticsearch Cluster · GitHub

I also tried

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.disable_allocation": false}'

and

curl -XPUT $(hostname -f):9200/_settings -d '
{"index.routing.allocation.enable": "all"}'

and removing one node from the cluster, which seemed to help, because
it finally assigned the shards to a node.

Unfortunately the same problem appeared again the next day, when the
cluster tried to create new indices.

Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy

I would like to find out the reason why the cluster doesn't assign
these new shards to any node or find a way to issue a command to the
cluster to reassign/redistribute all unassigned shards to a node.

Thanks in advance!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c7b24957-7c97-4f2f-b1b1-70b61cb29669%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/821de798-ba3e-4d36-9373-292344a95217%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.