Unassigned dangling_index_imported

Elastic Stack Elasticsearch
1

I have a cluster with status "red"

$ curl -s http://127.0.0.1:9200/_cluster/health | jq .
{
  "active_shards_percent_as_number": 99.90479817212491,
  "task_max_waiting_in_queue_millis": 0,
  "number_of_in_flight_fetch": 0,
  "number_of_pending_tasks": 0,
  "delayed_unassigned_shards": 0,
  "unassigned_shards": 4,
  "initializing_shards": 1,
  "cluster_name": "adzunalogs",
  "status": "red",                                  <=========== oops
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 2624,
  "active_shards": 5247,
  "relocating_shards": 1
}

and some indexes with UNASSIGNED DANGLING_INDEX_IMPORTED 

curl -s -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
logstash-2016.08.23 3 p UNASSIGNED DANGLING_INDEX_IMPORTED 
logstash-2016.08.23 3 r UNASSIGNED DANGLING_INDEX_IMPORTED 
logstash-2016.08.23 4 p UNASSIGNED DANGLING_INDEX_IMPORTED 
logstash-2016.08.23 4 r UNASSIGNED DANGLING_INDEX_IMPORTED

All other are "green"

curl -s http://127.0.0.1:9200/_cluster/health?level=indices | jq .
...
    "logstash-2016.08.23": {
      "unassigned_shards": 4,
      "initializing_shards": 0,
      "relocating_shards": 0,
      "active_shards": 6,
      "active_primary_shards": 3,
      "number_of_replicas": 1,
      "number_of_shards": 5,
      "status": "red"
    },
...

More details on the failing indices

$ curl -s -XGET localhost:9200/_cluster/health?level=shards | jq . | less


    "logstash-2016.08.23": {
      "shards": {
        "4": {
          "unassigned_shards": 2,
          "initializing_shards": 0,
          "relocating_shards": 0,
          "active_shards": 0,
          "primary_active": false,
          "status": "red"
        },
        "3": {
          "unassigned_shards": 2,
          "initializing_shards": 0,
          "relocating_shards": 0,
          "active_shards": 0,
          "primary_active": false,
          "status": "red"
        },

$ curl -s 'localhost:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host          ip            node       
  1749          2tb     2.3tb      1.1tb      3.5tb           66 IP1   IP1   node1   
  1749        1.8tb       2tb      1.4tb      3.5tb           58 IP2    IP2    node2
  1750        1.8tb       2tb      1.4tb      3.5tb           58 IP3    IP3 node3
     4                                                                                       UNASSIGNED

What would be the process of further debugging this?

In Cluster Health | Elasticsearch: The Definitive Guide [2.x] | Elastic I read

Once you know the index that is having problems, other APIs that we discuss in this chapter will tend to be more helpful.

but I am not sure where to look at for the other APIs.

How to resolve the unassigned shards says that rerouting can fix this but I am not sure if there are any implications.

Edit1: fixed the quote
Edit2: add health?level=shards output

2

After restarting all the cluster nodes one by one, in the end I still got those 4 unassigned shards.

I followed the instructions in How to resolve the unassigned shards but failed with 400

$ curl -XPOST 'http://127.0.0.1:9200/_cluster/reroute?pretty' -H 'Content-Type: application/json' -d'
> {
> "commands" : [
> {
> "allocate_stale_primary" : {
> "index" : "logstash-2016.08.23",
> "shard" : 4,
> "node" : "my_node_name",
> "accept_data_loss" : true
> }
> }
> ]
> }
> '
{
  "error" : {
    "root_cause" : [ {
      "type" : "illegal_argument_exception",
      "reason" : "No allocation command factory registered for name [allocate_stale_primary]"
    } ],
    "type" : "illegal_argument_exception",
    "reason" : "No allocation command factory registered for name [allocate_stale_primary]"
  },
  "status" : 400
}

I also tried to use the id of the node, not its node name but the result was the same, from

$ curl -s http://localhost:9200/_nodes?pretty
...
    "3MItGKMuSXiw1YxVIVwKig" : {
      "name" : "my_node_name",
      "transport_address" : "IP:9300",
...
3

I finally made ES "green" with following this https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/

$ curl -XPOST 'http://127.0.0.1:9200/_cluster/reroute' -d '
{
    "commands" : [
       {
           "allocate" : {
               "index" : "logstash-2016.08.23", "shard" : 4,
               "node": "my_node_name", "allow_primary": "true"
           }
       }
    ]
}
'

curl -XPOST 'http://127.0.0.1:9200/_cluster/reroute' -d '
{
    "commands" : [
       {
           "allocate" : {
               "index" : "logstash-2016.08.23", "shard" : 3,
               "node": "my_node_name", "allow_primary": "true"
           }
       }
    ]
}
'

but I was wondering if anyone can shed any light to the above.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.