ElasticSearch cluster allocation fails - disk threshold not met

Elastic Stack Elasticsearch
1

Hello community

I have ES1.6 cluster on Centos 6 and 7 combined, 4 node cluster at this moment.

  "cluster_name" : "nolo-elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 1961,
  "active_shards" : 1961,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 15,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0

Nodes:

rdsh157 10.0.22.157 30 28 0.01 d - rdsh157
aoes160   10.0.22.160 12 17 0.00 d m aoes160
rdsh158 10.0.22.158  6 11 0.01 d m rdsh158
aoes161   10.0.22.161  3 11 0.00 d * aoes161

In the log, the current master say

[2016-05-24 09:41:42,021][WARN ][cluster.routing.allocation.decider] [aoes161] After allocating, node [9g420bv0SBC8lbr3lAmwcw] would have more than the allowed 10% free disk threshold (4.1% free), preventing allocation

But the node [9g420bv0SBC8lbr3lAmwcw] does not exist under this name (??)

All servers in the cluster has enough of disk space (only the stsh157 one is nearly full)

It looks like the cluster does not want to allocate shards to other nodes or somehow does not know about them properly..

I performed the
curl -XPUT localhost:9200/_cluster/settings -d '{"persistent": {"cluster.routing.allocation.enable": "all"}}'

...No luck.

Any idea what can be happening?

2

All right - issue solved. For others:

More details on reason why reallocation fails can be get by issuing

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
        "commands" : [ {
              "allocate" : {
                  "index" : "insert_test2", 
                  "shard" : "0", 
                  "node" : "aoes160", 
                  "allow_primary" : true
              }
            }
        ]
    }'

There you can see:

[NO(target node version [1.6.0] is older than source node version [1.6.2])

It was all about different minor ES versions - the data get reallocated from older to the newer one but never vice versa...

3

Yep, this is a lucene compatibility restriction.