Shards stuck on initialising


(eistconn) #1

I have a two node cluster and after restarting them the cluster does not
assign all the shards.

On both nodes:
elasticsearch 1.0.1
$ java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1~0.12.04.2)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Cluster health:
{
"cluster_name" : "elastic_bwn",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 540,
"active_shards" : 607,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 471
}

I am very new to elastic search, and even though I have Googled through
similar problems and search in this group, I haven't found an answer.

Any help much appreciated.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

If it's a replica, which it appears to be, then just drop the replicas on
the problem index to 0 and then readd them.
This is a quick and dirty fix, I'm not sure if there is a way to recover
the underlying shard as it.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 4 September 2014 07:56, eistconn david.mantock@gmail.com wrote:

I have a two node cluster and after restarting them the cluster does not
assign all the shards.

On both nodes:
elasticsearch 1.0.1
$ java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1~0.12.04.2)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Cluster health:
{
"cluster_name" : "elastic_bwn",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 540,
"active_shards" : 607,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 471
}

I am very new to elastic search, and even though I have Googled through
similar problems and search in this group, I haven't found an answer.

Any help much appreciated.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZXctNzZJiVyRchhM-46%2BcHnHQG7npo72db2t3uMu_0yA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(eistconn) #3

I have been searching around on how to do that, but firstly how can you
tell which is the problem index?

Or is there an easy way to drop all replicas - can you use wildcard for
indexes:

curl -XPUT 'localhost:9200/logstash-*/_settings' -d '
{
"index" : {
"number_of_replicas" : 0
} }

On Thursday, 4 September 2014 00:47:38 UTC+2, Mark Walkom wrote:

If it's a replica, which it appears to be, then just drop the replicas on
the problem index to 0 and then readd them.
This is a quick and dirty fix, I'm not sure if there is a way to recover
the underlying shard as it.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 4 September 2014 07:56, eistconn <david....@gmail.com <javascript:>>
wrote:

I have a two node cluster and after restarting them the cluster does not
assign all the shards.

On both nodes:
elasticsearch 1.0.1
$ java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7)
(7u55-2.4.7-1ubuntu1~0.12.04.2)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Cluster health:
{
"cluster_name" : "elastic_bwn",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 540,
"active_shards" : 607,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 471
}

I am very new to elastic search, and even though I have Googled through
similar problems and search in this group, I haven't found an answer.

Any help much appreciated.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a25b48fc-b6fb-45bb-841a-720a0fd65e86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


#4

This may give you some insight into what is going on: curl -XGET 'localhost:9200/_cat/recovery?v'.

On Sep 3, 2014, at 3:58 PM, eistconn david.mantock@gmail.com wrote:

I have been searching around on how to do that, but firstly how can you tell which is the problem index?

Or is there an easy way to drop all replicas - can you use wildcard for indexes:

curl -XPUT 'localhost:9200/logstash-*/_settings' -d '
{
"index" : {
"number_of_replicas" : 0
} }

On Thursday, 4 September 2014 00:47:38 UTC+2, Mark Walkom wrote:
If it's a replica, which it appears to be, then just drop the replicas on the problem index to 0 and then readd them.
This is a quick and dirty fix, I'm not sure if there is a way to recover the underlying shard as it.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 4 September 2014 07:56, eistconn david....@gmail.com wrote:
I have a two node cluster and after restarting them the cluster does not assign all the shards.

On both nodes:
elasticsearch 1.0.1
$ java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1~0.12.04.2)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Cluster health:
{
"cluster_name" : "elastic_bwn",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 540,
"active_shards" : 607,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 471
}

I am very new to elastic search, and even though I have Googled through similar problems and search in this group, I haven't found an answer.

Any help much appreciated.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a25b48fc-b6fb-45bb-841a-720a0fd65e86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/51A07A6A-C62B-4A5E-B861-D2F60227FFEF%40elasticsearch.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #5

Yes you can use wildcards.

The easiest way to tell would be to use the cat APIs, eg _cat/shards?v

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 4 September 2014 08:58, eistconn david.mantock@gmail.com wrote:

I have been searching around on how to do that, but firstly how can you
tell which is the problem index?

Or is there an easy way to drop all replicas - can you use wildcard for
indexes:

curl -XPUT 'localhost:9200/logstash-*/_settings' -d '
{
"index" : {

    "number_of_replicas" : 0

} }

On Thursday, 4 September 2014 00:47:38 UTC+2, Mark Walkom wrote:

If it's a replica, which it appears to be, then just drop the replicas on
the problem index to 0 and then readd them.
This is a quick and dirty fix, I'm not sure if there is a way to recover
the underlying shard as it.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 4 September 2014 07:56, eistconn david....@gmail.com wrote:

I have a two node cluster and after restarting them the cluster does not
assign all the shards.

On both nodes:
elasticsearch 1.0.1
$ java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1~0.12.04.
2)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Cluster health:
{
"cluster_name" : "elastic_bwn",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 540,
"active_shards" : 607,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 471
}

I am very new to elastic search, and even though I have Googled through
similar problems and search in this group, I haven't found an answer.

Any help much appreciated.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a25b48fc-b6fb-45bb-841a-720a0fd65e86%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a25b48fc-b6fb-45bb-841a-720a0fd65e86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zo%2BV%3Dutx2LA6WoHgHcjqrhGdy9njEOvrbaj9%3Db5pU%3DEw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(eistconn) #6

Just did this:

curl -XPUT 'localhost:9200/_settings' -d ' {"index.routing.allocation.disable_allocation": false}'

Seems to have done the trick. Shards are now being assigned.

On Thursday, 4 September 2014 01:03:15 UTC+2, Mark Walkom wrote:

Yes you can use wildcards.

The easiest way to tell would be to use the cat APIs, eg _cat/shards?v

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 4 September 2014 08:58, eistconn <david....@gmail.com <javascript:>>
wrote:

I have been searching around on how to do that, but firstly how can you
tell which is the problem index?

Or is there an easy way to drop all replicas - can you use wildcard for
indexes:

curl -XPUT 'localhost:9200/logstash-*/_settings' -d '

{
"index" : {

    "number_of_replicas" : 0


} }

On Thursday, 4 September 2014 00:47:38 UTC+2, Mark Walkom wrote:

If it's a replica, which it appears to be, then just drop the replicas
on the problem index to 0 and then readd them.
This is a quick and dirty fix, I'm not sure if there is a way to recover
the underlying shard as it.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 4 September 2014 07:56, eistconn david....@gmail.com wrote:

I have a two node cluster and after restarting them the cluster does
not assign all the shards.

On both nodes:
elasticsearch 1.0.1
$ java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7)
(7u55-2.4.7-1ubuntu1~0.12.04.2)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Cluster health:
{
"cluster_name" : "elastic_bwn",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 540,
"active_shards" : 607,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 471
}

I am very new to elastic search, and even though I have Googled through
similar problems and search in this group, I haven't found an answer.

Any help much appreciated.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a25b48fc-b6fb-45bb-841a-720a0fd65e86%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a25b48fc-b6fb-45bb-841a-720a0fd65e86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/97b0e80d-db9e-446b-b10f-b5679d532e51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #7

Oh right, yeah that'd fix it too :stuck_out_tongue:

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 4 September 2014 09:22, eistconn david.mantock@gmail.com wrote:

Just did this:

curl -XPUT 'localhost:9200/_settings' -d ' {"index.routing.allocation.disable_allocation": false}'

Seems to have done the trick. Shards are now being assigned.

On Thursday, 4 September 2014 01:03:15 UTC+2, Mark Walkom wrote:

Yes you can use wildcards.

The easiest way to tell would be to use the cat APIs, eg _cat/shards?v

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 4 September 2014 08:58, eistconn david....@gmail.com wrote:

I have been searching around on how to do that, but firstly how can
you tell which is the problem index?

Or is there an easy way to drop all replicas - can you use wildcard for
indexes:

curl -XPUT 'localhost:9200/logstash-*/_settings' -d '

{
"index" : {

    "number_of_replicas" : 0



} }

On Thursday, 4 September 2014 00:47:38 UTC+2, Mark Walkom wrote:

If it's a replica, which it appears to be, then just drop the replicas
on the problem index to 0 and then readd them.
This is a quick and dirty fix, I'm not sure if there is a way to
recover the underlying shard as it.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 4 September 2014 07:56, eistconn david....@gmail.com wrote:

I have a two node cluster and after restarting them the cluster does
not assign all the shards.

On both nodes:
elasticsearch 1.0.1
$ java -version
java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7)
(7u55-2.4.7-1ubuntu1~0.12.04.2)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Cluster health:
{
"cluster_name" : "elastic_bwn",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 540,
"active_shards" : 607,
"relocating_shards" : 0,
"initializing_shards" : 2,
"unassigned_shards" : 471
}

I am very new to elastic search, and even though I have Googled
through similar problems and search in this group, I haven't found an
answer.

Any help much appreciated.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/8f257a81-5f16-4e7c-b0bb-9a580a198412%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a25b48fc-b6fb-45bb-841a-720a0fd65e86%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a25b48fc-b6fb-45bb-841a-720a0fd65e86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/97b0e80d-db9e-446b-b10f-b5679d532e51%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/97b0e80d-db9e-446b-b10f-b5679d532e51%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YjPW636rjOBrzixL1vr7m6VpNe4cZQxL6MF5nWMuczpA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Malini) #8

I have 3 node cluster with two replicas. Our cluster was running fine and I had to restart elasticsearch.
After the restart I am saw 2 unassigned shards.
So I ran the following command
curl -XPUT 'localhost:9200/_settings' \ -d '{"index.routing.allocation.disable_allocation": false}'

which assigned all the shards.

What does that command mean? Should I do this if I see unassigned shards? Would this have any affect on future indicies etc?

Thanks
-Malini


(system) #9