Stuck with unassigned shards

Elastic Stack Elasticsearch
1

Hello,

Following a forced reboot we're stuck with 117 unassigned shards, and it doesn't move from 2 hours now.

What can we do?

Cluster Health:

{
  "cluster_name" : "ELASTIC",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 6,
  "number_of_data_nodes" : 6,
  "active_primary_shards" : 4623,
  "active_shards" : 9129,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 117,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 98.73458792991563
}

Cluster Settings:

{
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"
        }
      }
    },
    "discovery" : {
      "zen" : {
        "minimum_master_nodes" : "3"
      }
    },
    "indices" : {
      "breaker" : {
        "fielddata" : {
          "limit" : "60%"
        }
      }
    }
  },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "cluster_concurrent_rebalance" : "12",
          "node_concurrent_recoveries" : "6",
          "node_initial_primaries_recoveries" : "6",
          "enable" : "all"
        }
      }
    },
    "indices" : {
      "recovery" : {
        "concurrent_streams" : "12",
        "translog_size" : "1024kb",
        "max_bytes_per_sec" : "500mb",
        "file_chunk_size" : "1024kb"
      }
    }
  }
}

Unasigned shards are only replicas.

What to do to make them allocated?

ES version:

  • 5 nodes at 2.3.2
  • 1 node at 2.4.0

Thanks
Bruno

2

All nodes in the cluster must be on the same version. The only time you should have clusters with mixed nodes is during a rolling upgrade. Shards allocated to a node with a newer version can not be copied or transferred to a node with an older version if the Lucene version differs. Are the indices with missing replicas by any chance all allocated to the 2.4.0 node?

As you already have some shards on a 2.4.0 node, I would recommend upgrading all the other nodes to 2.4.0 as well.

3

yes, thanks, I just saw that pattern...