Tweezer fixes to status-red don't work, may need sledgehammer

pitaga1 · November 21, 2014, 7:21pm

When I shutdown and restart Elasticsearch on my Ubuntu laptop, the status
is yellow until I attempt to open an index, at which time the status
changes to red. The output for

curl "http://localhost:9200/_cluster/health?pretty=true"

is

{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 5,
  "unassigned_shards" : 7
}

I've tried deleting all but one index, shutting down and restarting. I've
also tried issuing

curl -XPUT 'localhost:9200/testindex2/_settings' -d

'{"index.routing.allocation.disable_allocation": false}'

shutting down and restarting. I've also tried issuing

curl XPOST 'http://localhost:9200/_cluster/reroute?pretty=true' -d
'{"commands" : [ { "allocate" : { "index" : "testindex1", "shard" : 4 ,
"node" : "gdVC517DR4OiE0v44_1n5A", "allow_primary" : 1 } }]}’

for one particular index and shard, with no apparent effect on the output of

curl -XGET http://localhost:9200/_cluster/state?pretty=true

In each case, I shutdown with

curl -XPOST 'http://localhost:9200/_shutdown'

and restarted with

/etc/init.d/elasticsearch restart

The one remaining undeleted index is currently used only for
experimentation. I can delete and then re-establish it if necessary. Or I
can apply a bigger sledgehammer if necessary.

I found elements of diagnostics and suggested solutions by searching the
web. I am not now, and possibly I will never be, a qualified Elasticsearch
administrator. I'd be grateful for suggestions on how to fix this, in terms
a child can understand.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a809850-58d0-4ee8-803c-18160088a2ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

brian_yoder · November 24, 2014, 4:15pm

Just a wild guess, but it seems that the /etc/init.d/elasticsearch restart
command will, if properly named, stop a currently running instance and then
start it.

If you issue the curl _shutdown command and then the restart command
directly after without any delays, then perhaps that double blow from your
sledgehammer is causing some corruption.

In general, it's not good to mix HTTP REST (curl) commands and scripts that
directly handle processes without adequate delays to ensure they aren't
hammering on each other.

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d2679372-8184-4578-a9f6-a88dfe31ad42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

pitaga1 · November 24, 2014, 6:08pm

Brian, thanks for the help on correcting what I've been doing. Once I get
my system unwedged, I won't mix curl commands and scripts.

I deleted the one remaining index that I had established, and then all the
remaining .marvel indexes. Then I re-established mapping and settings for
my index. When I tried to open the index, I got an '"acknowledged" : false'
response on the first try, and an '"acknowledged" : true' response on the
second try. When I tried to put a document in this index, I got

"error":"UnavailableShardsException[[testindex2][3] [2] shardIt, [0]

active : Timeout waiting for [1m] ...

According to what I've read, even if I uninstall and re-install
Elasticsearch on my laptop, it might be unusable. I expect a need a tweezer
fix that's informed by knowledge of how Elasticsearch manages shards. I'd
be grateful for possible recipes on how to fix this, and also for a pointer
to a clear and concise description of Elasticsearch shard management.

Thanks.

On Friday, November 21, 2014 11:21:26 AM UTC-8, Pitaga wrote:

When I shutdown and restart Elasticsearch on my Ubuntu laptop, the status
is yellow until I attempt to open an index, at which time the status
changes to red. The output for
curl "http://localhost:9200/_cluster/health?pretty=true"
is
{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 5,
  "unassigned_shards" : 7
}
I've tried deleting all but one index, shutting down and restarting. I've
also tried issuing
curl -XPUT 'localhost:9200/testindex2/_settings' -d 
'{"index.routing.allocation.disable_allocation": false}'

shutting down and restarting. I've also tried issuing

curl XPOST 'http://localhost:9200/_cluster/reroute?pretty=true' -d
'{"commands" : [ { "allocate" : { "index" : "testindex1", "shard" : 4 ,
"node" : "gdVC517DR4OiE0v44_1n5A", "allow_primary" : 1 } }]}’

for one particular index and shard, with no apparent effect on the output
of
curl -XGET http://localhost:9200/_cluster/state?pretty=true
In each case, I shutdown with
curl -XPOST 'http://localhost:9200/_shutdown'
and restarted with
/etc/init.d/elasticsearch restart
The one remaining undeleted index is currently used only for
experimentation. I can delete and then re-establish it if necessary. Or I
can apply a bigger sledgehammer if necessary.

I found elements of diagnostics and suggested solutions by searching the
web. I am not now, and possibly I will never be, a qualified Elasticsearch
administrator. I'd be grateful for suggestions on how to fix this, in terms
a child can understand.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8f4a5795-eec6-4415-acef-16d8bc29e63b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · November 24, 2014, 9:44pm

You have 7 unassigned shards. Are you sure you always had one node in the
cluster?

It seems there were once more than one nodes. So start two (or more) nodes
on your laptop and the cluster state will get green.

Note, on single node, always create with replica level 0. The default is 1,
which has been source of confusion, it is convenient to show fault
tolerance, but on a laptop, that does not make sense.

Jörg

On Mon, Nov 24, 2014 at 7:08 PM, Pitaga achats@blarg.net wrote:

Brian, thanks for the help on correcting what I've been doing. Once I get
my system unwedged, I won't mix curl commands and scripts.

I deleted the one remaining index that I had established, and then all the
remaining .marvel indexes. Then I re-established mapping and settings for
my index. When I tried to open the index, I got an '"acknowledged" : false'
response on the first try, and an '"acknowledged" : true' response on the
second try. When I tried to put a document in this index, I got
"error":"UnavailableShardsException[[testindex2][3] [2] shardIt, [0]
active : Timeout waiting for [1m] ...

According to what I've read, even if I uninstall and re-install
Elasticsearch on my laptop, it might be unusable. I expect a need a tweezer
fix that's informed by knowledge of how Elasticsearch manages shards. I'd
be grateful for possible recipes on how to fix this, and also for a pointer
to a clear and concise description of Elasticsearch shard management.

Thanks.

On Friday, November 21, 2014 11:21:26 AM UTC-8, Pitaga wrote:
When I shutdown and restart Elasticsearch on my Ubuntu laptop, the
status is yellow until I attempt to open an index, at which time the status
changes to red. The output for
curl "http://localhost:9200/_cluster/health?pretty=true"
is
{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 5,
  "unassigned_shards" : 7
}
I've tried deleting all but one index, shutting down and restarting. I've
also tried issuing
curl -XPUT 'localhost:9200/testindex2/_settings' -d
'{"index.routing.allocation.disable_allocation": false}'

shutting down and restarting. I've also tried issuing

curl XPOST 'http://localhost:9200/_cluster/reroute?pretty=true' -d
'{"commands" : [ { "allocate" : { "index" : "testindex1", "shard" : 4 ,
"node" : "gdVC517DR4OiE0v44_1n5A", "allow_primary" : 1 } }]}’

for one particular index and shard, with no apparent effect on the output
of
curl -XGET http://localhost:9200/_cluster/state?pretty=true
In each case, I shutdown with
curl -XPOST 'http://localhost:9200/_shutdown'
and restarted with
/etc/init.d/elasticsearch restart
The one remaining undeleted index is currently used only for
experimentation. I can delete and then re-establish it if necessary. Or I
can apply a bigger sledgehammer if necessary.

I found elements of diagnostics and suggested solutions by searching the
web. I am not now, and possibly I will never be, a qualified Elasticsearch
administrator. I'd be grateful for suggestions on how to fix this, in terms
a child can understand.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8f4a5795-eec6-4415-acef-16d8bc29e63b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8f4a5795-eec6-4415-acef-16d8bc29e63b%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFvTbdPazd4bWOw5wj_9m%3D9DrK%2BCUMTPR5qEQVy99UJ0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

pitaga1 · November 24, 2014, 11:03pm

Thanks, Joerg.

bin/elasticsearch -Des.node.data=true -Des.de.master=false

-Des.node.name=DataOne

and

curl -XPUT 'localhost:9200/testindex2/_settings' -d ' {"index" :

{"number_of_replicas" : 0}}'

with several restarts, have got me to a state where I see the following
command-response sequence:

curl "http://localhost:9200/_cluster/health?pretty=true"{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 33
}
curl -XPOST "http://localhost:9200/testindex2/_open"
{"acknowledged":false}
curl -XPOST "http://localhost:9200/testindex2/_open"
{"acknowledged":true}
curl "http://localhost:9200/_cluster/health?pretty=true"{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1,
  "active_shards" : 1,
  "relocating_shards" : 0,
  "initializing_shards" : 5,
  "unassigned_shards" : 33
}

At one point I saw
curl "http://localhost:9200/_cluster/health?pretty=true" {
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 17,
"active_shards" : 17,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 17
}

Marvel overview now shows a single node at 192.168.0.9:9300. That's 9300,
not 9200. I have no idea how the second node disappeared. It looks like
when I try to open testindex2, after deleting it and partially
re-establishing it, Elasticsearch gets stuck while trying to initialize 5
residual shards.

I'm utterly confused. I'm also motivated to learn more about node
management. I'd really like to do this with a working installation. How can
I untangle myself?

Thanks.

On Friday, November 21, 2014 11:21:26 AM UTC-8, Pitaga wrote:

When I shutdown and restart Elasticsearch on my Ubuntu laptop, the status
is yellow until I attempt to open an index, at which time the status
changes to red. The output for
curl "http://localhost:9200/_cluster/health?pretty=true"
is
{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 5,
  "unassigned_shards" : 7
}
I've tried deleting all but one index, shutting down and restarting. I've
also tried issuing
curl -XPUT 'localhost:9200/testindex2/_settings' -d 
'{"index.routing.allocation.disable_allocation": false}'

shutting down and restarting. I've also tried issuing

curl XPOST 'http://localhost:9200/_cluster/reroute?pretty=true' -d
'{"commands" : [ { "allocate" : { "index" : "testindex1", "shard" : 4 ,
"node" : "gdVC517DR4OiE0v44_1n5A", "allow_primary" : 1 } }]}’

for one particular index and shard, with no apparent effect on the output
of
curl -XGET http://localhost:9200/_cluster/state?pretty=true
In each case, I shutdown with
curl -XPOST 'http://localhost:9200/_shutdown'
and restarted with
/etc/init.d/elasticsearch restart
The one remaining undeleted index is currently used only for
experimentation. I can delete and then re-establish it if necessary. Or I
can apply a bigger sledgehammer if necessary.

I found elements of diagnostics and suggested solutions by searching the
web. I am not now, and possibly I will never be, a qualified Elasticsearch
administrator. I'd be grateful for suggestions on how to fix this, in terms
a child can understand.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/15cd7713-a78c-4727-9de3-664837034b07%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

pitaga1 · November 25, 2014, 10:36pm

Doing a thorough job of deleting all indexes got my installation back to
status yellow, apparently fully functional when I re-established my test
index with a new name. Sledgehammer, but I didn't need to sow the ground
with salt.

On Friday, November 21, 2014 11:21:26 AM UTC-8, Pitaga wrote:

When I shutdown and restart Elasticsearch on my Ubuntu laptop, the status
is yellow until I attempt to open an index, at which time the status
changes to red. The output for
curl "http://localhost:9200/_cluster/health?pretty=true"
is
{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 5,
  "unassigned_shards" : 7
}
I've tried deleting all but one index, shutting down and restarting. I've
also tried issuing
curl -XPUT 'localhost:9200/testindex2/_settings' -d 
'{"index.routing.allocation.disable_allocation": false}'

shutting down and restarting. I've also tried issuing

curl XPOST 'http://localhost:9200/_cluster/reroute?pretty=true' -d
'{"commands" : [ { "allocate" : { "index" : "testindex1", "shard" : 4 ,
"node" : "gdVC517DR4OiE0v44_1n5A", "allow_primary" : 1 } }]}’

for one particular index and shard, with no apparent effect on the output
of
curl -XGET http://localhost:9200/_cluster/state?pretty=true
In each case, I shutdown with
curl -XPOST 'http://localhost:9200/_shutdown'
and restarted with
/etc/init.d/elasticsearch restart
The one remaining undeleted index is currently used only for
experimentation. I can delete and then re-establish it if necessary. Or I
can apply a bigger sledgehammer if necessary.

I found elements of diagnostics and suggested solutions by searching the
web. I am not now, and possibly I will never be, a qualified Elasticsearch
administrator. I'd be grateful for suggestions on how to fix this, in terms
a child can understand.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/44983fbf-dc7d-498a-9e34-387124fa3225%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · November 25, 2014, 10:44pm

You can get it to green if you get rid of the replicas - curl -XPUT
localhost:9200/*/_settings -d '{ "index" : { "number_of_replicas" : 0 } }'

On 26 November 2014 at 09:36, Pitaga achats@blarg.net wrote:

Doing a thorough job of deleting all indexes got my installation back to
status yellow, apparently fully functional when I re-established my test
index with a new name. Sledgehammer, but I didn't need to sow the ground
with salt.

On Friday, November 21, 2014 11:21:26 AM UTC-8, Pitaga wrote:
When I shutdown and restart Elasticsearch on my Ubuntu laptop, the
status is yellow until I attempt to open an index, at which time the status
changes to red. The output for
curl "http://localhost:9200/_cluster/health?pretty=true"
is
{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 5,
  "unassigned_shards" : 7
}
I've tried deleting all but one index, shutting down and restarting. I've
also tried issuing
curl -XPUT 'localhost:9200/testindex2/_settings' -d
'{"index.routing.allocation.disable_allocation": false}'

shutting down and restarting. I've also tried issuing

curl XPOST 'http://localhost:9200/_cluster/reroute?pretty=true' -d
'{"commands" : [ { "allocate" : { "index" : "testindex1", "shard" : 4 ,
"node" : "gdVC517DR4OiE0v44_1n5A", "allow_primary" : 1 } }]}’

for one particular index and shard, with no apparent effect on the output
of
curl -XGET http://localhost:9200/_cluster/state?pretty=true
In each case, I shutdown with
curl -XPOST 'http://localhost:9200/_shutdown'
and restarted with
/etc/init.d/elasticsearch restart
The one remaining undeleted index is currently used only for
experimentation. I can delete and then re-establish it if necessary. Or I
can apply a bigger sledgehammer if necessary.

I found elements of diagnostics and suggested solutions by searching the
web. I am not now, and possibly I will never be, a qualified Elasticsearch
administrator. I'd be grateful for suggestions on how to fix this, in terms
a child can understand.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/44983fbf-dc7d-498a-9e34-387124fa3225%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/44983fbf-dc7d-498a-9e34-387124fa3225%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DOw80b9R%3Dt8edySUf_pZyxH4N83axkTaStoPOaEn%2BHiQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Status: Red on ES Cluster Elasticsearch	8	539	April 13, 2018
Cluster Health status is Red Elasticsearch	4	1015	October 14, 2021
Red status elastic Elasticsearch	10	1469	December 6, 2017
Why Cluster Status is Red, please help Elasticsearch	9	1687	July 5, 2017
How to fix signal red? Elasticsearch	7	3656	February 26, 2020

Tweezer fixes to status-red don't work, may need sledgehammer

Related topics