Failed recovering a shard after cluster restart

Elastic Stack Elasticsearch
1

Hi,

I have an elasticSearch cluster of 4 nodes. After restarting each one of
them for some configuration changes, I got into the "red" state. I didn't
really understand what was happening and I restarted the cluster again.
After few minutes, I got back to "green" state but the number of documents
has dropped from ~10 000 000 to ~8 000 000. I tried to refresh the index or
to run a "_count" query, but I'm still getting the same result.

I looked into the logs and apparently, there is an issue with the
recovering of one shard.

Here are the detailed logs
: https://gist.github.com/lasryaric/7369e6c1cc3b9619457c
_cluster/health: https://gist.github.com/lasryaric/c248b7b3b381d4394f77
_nodes?all : https://gist.github.com/lasryaric/afb25863ed49fdfc864d

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hey

you can out of memory on that node (from then on its behaviour is basically
undefined). Do your nodes have a different setup? Just wondering why it did
not happen on other nodes...

On Wed, Jun 12, 2013 at 5:03 PM, aric lasry lasry.aric@gmail.com wrote:

Hi,

I have an elasticSearch cluster of 4 nodes. After restarting each one of
them for some configuration changes, I got into the "red" state. I didn't
really understand what was happening and I restarted the cluster again.
After few minutes, I got back to "green" state but the number of documents
has dropped from ~10 000 000 to ~8 000 000. I tried to refresh the index or
to run a "_count" query, but I'm still getting the same result.

I looked into the logs and apparently, there is an issue with the
recovering of one shard.

Here are the detailed logs :
es_failed_shard.log · GitHub
_cluster/health: host:9200/_cluster/health · GitHub
_nodes?all : host:9200/_nodes?all · GitHub

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.