Unassigned shards after updating # of replica

Hi,
I am using ES 0.90.13 with 10 clusters. Java Sun 1.7.53.

We find out that the # of replicas on our ES clusters was zero.
Hence, we increased the # of replica last week on ES2 to 1.
Since then the cluster became yellow <-> green. It runs fine for several
hours and then became yellow (with X unassigned shards). After few hours it
became green.

Most of the time it became green, but from time few time it is stacked on 2
unassigned shards.

From the logs:

[index.shard.service ] [es2-s01] [ds77777][3] suspect illegal state: trying
to move shard from primary mode to replica mode

[2014-08-11 00:33:10,384][WARN ][cluster.action.shard ] [es2-s02]
[ds6765431_1_s][1] sending failed shard for [ds6765431_1_s][1],
node[M34ta1aGTv-S4ahkrXK5DA], [R], s[STARTED], indexUUID
[4T8zV8r7RUu8UkvQYsoz3w], reason [master
[es2-s03][4rJC0nkVR1SgZhAcOK2MZw][inet[/10.1.112.3:9300]]{data_center=ny4,
max_local_storage_nodes=1, master=true} marked shard as started, but shard
has not been created, mark shard as failed]

This is a producation cluster. Please assist.
Yaniv

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d2934403-9c04-42bd-9259-68313c25e1f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.