ElasticSearch:Failed to execute phase [dfs], all shards failed

I am using elastic search 2.3.5 and when i started contacting it from my Java Code i am getting the following Exception:

Failed to execute phase [dfs], all shards failedat org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:129)
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:115)

Here is what i am seeing in elastic search console. I have set the following in bin/elasticsearch

export ES_HEAP_SIZE=10g
echo "ES_HEAP_SIZE=[$ES_HEAP_SIZE]"

Here is my yml file

cluster.name: mycluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: hdp1

node.master: true
node.data: true
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/es
#
# Path to log files:
#
path.logs: /data/es/logs

path.plugins: /data/es/plugins
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.mlockall: true

Appreciate any help.

Here is what i am getting for cluster health:

{
   "cluster_name": "mycluster",
   "status": "red",
   "timed_out": false,
   "number_of_nodes": 1,
   "number_of_data_nodes": 1,
   "active_primary_shards": 0,
   "active_shards": 0,
   "relocating_shards": 0,
   "initializing_shards": 0,
   "unassigned_shards": 54,
   "delayed_unassigned_shards": 0,
   "number_of_pending_tasks": 0,
   "number_of_in_flight_fetch": 0,
   "task_max_waiting_in_queue_millis": 0,
   "active_shards_percent_as_number": 0,
   "indices": {
    "index1": {
     "status": "red",
     "number_of_shards": 3,
     "number_of_replicas": 2,
     "active_primary_shards": 0,
     "active_shards": 0,
     "relocating_shards": 0,
     "initializing_shards": 0,
     "unassigned_shards": 9
  }
}

It looks like there are 54 unassigned shards. I am not sure how to make the shards get assigned to a node.

I tried starting another instance (with different node name) and it this is what i got for helath status:

{
   "cluster_name": "mycluster",
   "status": "red",
   "timed_out": false,
   "number_of_nodes": 2,
   "number_of_data_nodes": 2,
   "active_primary_shards": 0,
   "active_shards": 0,
   "relocating_shards": 0,
   "initializing_shards": 0,
   "unassigned_shards": 54,
   "delayed_unassigned_shards": 0,
   "number_of_pending_tasks": 0,
   "number_of_in_flight_fetch": 0,
   "task_max_waiting_in_queue_millis": 0,
   "active_shards_percent_as_number": 0,
   "indices": {
  "index1": {
     "status": "red",
     "number_of_shards": 3,
     "number_of_replicas": 2,
     "active_primary_shards": 0,
     "active_shards": 0,
     "relocating_shards": 0,
     "initializing_shards": 0,
     "unassigned_shards": 9,
     "shards": {
        "0": {
           "status": "red",
           "primary_active": false,
           "active_shards": 0,
           "relocating_shards": 0,
           "initializing_shards": 0,
           "unassigned_shards": 3
        },
        "1": {
           "status": "red",
           "primary_active": false,
           "active_shards": 0,
           "relocating_shards": 0,
           "initializing_shards": 0,
           "unassigned_shards": 3
        },
        "2": {
           "status": "red",
           "primary_active": false,
           "active_shards": 0,
           "relocating_shards": 0,
           "initializing_shards": 0,
           "unassigned_shards": 3
        }
     }
  }
}

Not sure what i am doing wrong.

Thanks

Hi,

Look at _cat/shards and you can see which shards are unassigned. Then you can check your data path (/data/es) and see if these shards exist. Its possible you lost the data or you need to start the node(s) that these shards exist on.