Hello
We have an elasticsearch cluster with 5 nodes used for graylog2, a log
centralization solution.
We have four nodes that hold data + 1 node that has node.data=false that is
used by the graylog server to index messages.
Today we restarted the node that does not hold any data and strangely that
caused all the shards to be re-initialized. However only part of the shards
have recovered, there are four shards stuck in initializing state whereas
four others(their replicas) are in an unassigned state.
The only error present on the nodes is this one
cat /var/log/elasticsearch/Graylog2.log| grep -B7 -i err
[2012-10-02 16:05:58,867][WARN ][index.engine.robin ] [server_name]
[graylog2][4] failed to flush after setting shard to inactive
org.elasticsearch.index.engine.EngineClosedException: [graylog2][4]
CurrentState[CLOSED]
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:799)
at
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:223)
at
org.elasticsearch.indices.memory.IndexingMemoryBufferController$ShardsIndicesStatusChecker.run(IndexingMemoryBufferController.java:147)
at
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:218)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
We are using index..number_of_replicas=1
Is there a way to get the unassigned shards to be reinitialized ?
Thanks
--