Seeing some recent problems on master where shortly after a cluster reboot
and full recovery (all shards green, all nodes connected), on occasion a
shard or two will start being marked as recovery (rebalancing is disabled,
so it's not due to it being moved around). After a few minutes they
sometimes say they're initializing (an order of magnitude after the amount
of time it should take to transfer the shard given all the throttling
settings and network), but in the end they'll never come back up and just
remain in that state until I manually reroute the shard or restart the
server.
We were seeing similar problems on a larger scale on 0.90.3 (multiple
shards never recovering, all of them replicas of the same primary). I
thought I saw something about that being fixed on github, but can't recall
the exact issue now. Any chance these are related? Is there something I can
do to debug what is going on while the shards say initializing/recovering?
Nothing is currently coming up in my logs.
anything in the logs, cluster state, hot_threads output, which might add
some information to this?
I am worried about the 'on occasion' part. If everything is good and you do
not change the amount of nodes in your cluster, there should be no need to
mark a shard to recover, so something has happened at that stage. Anything
in the logs then?
Seeing some recent problems on master where shortly after a cluster reboot
and full recovery (all shards green, all nodes connected), on occasion a
shard or two will start being marked as recovery (rebalancing is disabled,
so it's not due to it being moved around). After a few minutes they
sometimes say they're initializing (an order of magnitude after the amount
of time it should take to transfer the shard given all the throttling
settings and network), but in the end they'll never come back up and just
remain in that state until I manually reroute the shard or restart the
server.
We were seeing similar problems on a larger scale on 0.90.3 (multiple
shards never recovering, all of them replicas of the same primary). I
thought I saw something about that being fixed on github, but can't recall
the exact issue now. Any chance these are related? Is there something I can
do to debug what is going on while the shards say initializing/recovering?
Nothing is currently coming up in my logs.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.