Elastic search error in application, Error message - " An error occurred while requesting elastic search for data on index 'denmod_y_*'

Once the application runs & When elastic search for data, it was unable to retrieve. It says error while searching the data. It has two types of error 1) All shards failed error 2) Index not found exception.

We have rebooted all the master & elastic search servers but we are still unable to find the solution. Also im new to elastic search.

  1. Index not found exception
{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "resource.type": "index_or_alias",
        "resource.id": "denmod_y_104242",
        "index_uuid": "_na_",
        "index": "denmod_y_104242"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "resource.type": "index_or_alias",
    "resource.id": "denmod_y_104242",
    "index_uuid": "_na_",
    "index": "denmod_y_104242"
  },
  "status": 404
}
{
  "error": {
    "root_cause": [

    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [

    ],
    "caused_by": {
      "type": "no_shard_available_action_exception",
      "reason": null,
      "index_uuid": "Qtybg8VFStaOKuDkwJ00RA",
      "shard": "5",
      "index": "en_y_77"
    }
  },
  "status": 503
}
  1. For first error, we need the shards to be allocated or assigned 2) For second error, index error to be rectified.

What is the output of:

GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v
# Optionally 
GET /_cat/shards?v

If some outputs are too big, please share them on gist.github.com and link them here.

Please find the below output:

GET /_cat/nodes?v

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
168.124.25.122 19 67 0 di - elk-denmod-6
168.124.54.142 26 68 0 di - elk-denmod-5
168.124.29.126 25 68 0 di - elk-denmod-3
168.124.170.244 25 68 0 di - elk-denmod-4
168.124.25.140 13 66 0 di - elk-denmod-1
168.124.147.161 36 70 24 m * elk-denmod-web
168.124.29.129 24 67 1 di - elk-denmod-2

GET /_cat/health?v

epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1563176654 09:44:14 elk-denmod red 7 6 20 10 0 0 596 0 - 3.2%

Please read this about how to format.

Could you also provide the other outputs I asked for?

We don't have kibana to give the format like that, we use only cerebro tool elastic search.

And? Does it prevent you from running the queries I asked and format the result?

please find the below github link for the above outputs.

GitHub_Link

We can see that a lot of shards are unassigned. Which explains this red state and why you can't really search.

You have apparently 596 missing shards.

Could you run:

GET /_cluster/pending_tasks
GET /_cluster/allocation/explain?include_disk_info=true

GET /_cluster/pending_tasks --> No tasks

GET /_cluster/allocation/explain?include_disk_info=true