Cluster health becomes RED after create a new index

(Xudong You) #1

I met a very strange issue. My cluster (ES 1.7.1) always enters red status if I create a new index with only one doc.

Repro steps:

  1. get current health:

    1458814224 10:10:24 my-es-cluster green 12 6 36 36 0 0 0 0

    It is green.

  2. create a new index by posting a doc:

    "id": "1"

    then got following response:

"error": "UnavailableShardsException[[testindex][2] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: delete {[testindex][doc][1]}]"

Also, the cluster health becomes "red"

I checked the ES log file, only found following error message:
[2016-03-24 10:04:40,866][DEBUG][action.index ] [data-node-vm5] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]

How to troubleshoot the root cause?



May be that you should assign "by hand" your shard.
GET _cat/shards

And check if this setting is correctly setted

More here

(Xudong You) #3

Thanks iliasse!
You are right, the cluster.routing.allocation.enable was set to "none" accidentally by someone.
After changing it back to "all", I am able to create new index.

(Xudong You) #4

BTW: There is no any hint in Elasticsearch log why index creating failed. Is there any place to find detailed reason for such error?

(system) #5