Dangling indices

Elastic Stack Elasticsearch
1

Hi there,

For one of our applications I'm suffering with dangling indices that don't
seem to get assigned anymore. The cluster state is red, however there are
only 12 shards unassigned.

{"cluster_name":"my-cluster","status":"red","timed_out":false,"number_of_nodes":21,"number_of_data_nodes":3,"active_primary_shards":319,"active_shards":638,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":12}

This is now for at least an hour, and it is pretty strange as usually there
are either shard relocating or initializing when there are unassigned
shards. This is also under the state "yellow" instead of red.
Does anyone know what is wrong, and even better: how to resolve it?

Best regards,

Robin Verlangen
Chief Data Architect

W http://www.robinverlangen.nl
E robin@us2.nl

http://goo.gl/Lt7BC
What is CloudPelican? http://goo.gl/HkB3D

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADVHTB_tWddbREj6U3V_riOXk29MwWDQ6eJVHfvg%2BBMmUvAo6Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

I think I figured out why they didn't join. It seems that all of the shards
belonged to indices that were recently deleted asynchronously.

Is it possible that a node disconnect during a delete causes this kind of
behaviour? After a full cluster restarted, combined with "re-deleting" the
indices everything went back to status green.

Current ES version is 0.90.5

Best regards,

Robin Verlangen
Chief Data Architect

W http://www.robinverlangen.nl
E robin@us2.nl

http://goo.gl/Lt7BC
What is CloudPelican? http://goo.gl/HkB3D

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.

On Fri, Jan 3, 2014 at 7:44 PM, Robin Verlangen robin@us2.nl wrote:

Hi there,

For one of our applications I'm suffering with dangling indices that don't
seem to get assigned anymore. The cluster state is red, however there are
only 12 shards unassigned.

{"cluster_name":"my-cluster","status":"red","timed_out":false,"number_of_nodes":21,"number_of_data_nodes":3,"active_primary_shards":319,"active_shards":638,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":12}

This is now for at least an hour, and it is pretty strange as usually
there are either shard relocating or initializing when there are unassigned
shards. This is also under the state "yellow" instead of red.
Does anyone know what is wrong, and even better: how to resolve it?

Best regards,

Robin Verlangen
Chief Data Architect

W http://www.robinverlangen.nl
E robin@us2.nl

http://goo.gl/Lt7BC
What is CloudPelican? http://goo.gl/HkB3D

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADVHTB9ZDXLwKnc8ZkV5_%2Bu6oJG0AON6ORHkE90mC%3D4%2BpiQ%3D7A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

I can confirm that I saw this effect before. A second delete of an already
deleted index dropped it finally, after a node had OOM and could not
perform the delete.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEJe4jd-kGub7pSD7ReY%3DOvaLpiiYqs%2BW0-Y5_UJmfnpA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Ok, thank you for confirming. Do you also know whether there is an open
issue related to this? If not, I would be willing to file one, in order to
make sure it will prevent others from suffering the same issue.

Best regards,

Robin Verlangen
Chief Data Architect

W http://www.robinverlangen.nl
E robin@us2.nl

http://goo.gl/Lt7BC
What is CloudPelican? http://goo.gl/HkB3D

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.

On Fri, Jan 3, 2014 at 8:25 PM, joergprante@gmail.com <joergprante@gmail.com

wrote:

I can confirm that I saw this effect before. A second delete of an already
deleted index dropped it finally, after a node had OOM and could not
perform the delete.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEJe4jd-kGub7pSD7ReY%3DOvaLpiiYqs%2BW0-Y5_UJmfnpA%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADVHTB97LSobFcgS5BRTOSoPAP3UJAzP1v%2B%2B0RLUeLMexad4KQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

There was an issue regarding index deletion, back in 0.90.1, but I don't
know if this is related.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHhT25q9_WZEHNhJz0z%3DC%3DsLYx1GKyhvG5VLeY6TxkgsQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.