[elasticsearch 2.3] automatic shard rebalance?

[mobidyc]

Hello,

I have one node on 24 receiving zero shards, and I can't figure out why.
Nothing in the logs, cluster is green, node accepts requests.

any Idea on how to identify the problem?

# curl -s '127.0.0.1:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host          ip            node                
    49        123gb     1.4tb        2tb      3.5tb           41 192.168.25.46 192.168.25.46 node-24 
    50        146gb     1.3tb      2.2tb      3.5tb           36 192.168.25.45 192.168.25.45 node-23 
    50      131.5gb     1.3tb      2.2tb      3.5tb           37 192.168.25.42 192.168.25.42 node-20 
    49      150.5gb     1.5tb      1.9tb      3.5tb           44 192.168.25.35 192.168.25.35 node-13 
    49        161gb     1.3tb      2.2tb      3.5tb           37 192.168.25.44 192.168.25.44 node-22 
    49      130.1gb     1.3tb      2.2tb      3.5tb           37 192.168.25.43 192.168.25.43 node-21 
    50      830.2gb       2tb      1.5tb      3.5tb           57 192.168.25.17 192.168.25.17 node-03 
    49      129.7gb     1.3tb      2.2tb      3.5tb           37 192.168.25.40 192.168.25.40 node-18 
     0           0b     1.8tb      1.7tb      3.5tb           50 192.168.25.15 192.168.25.15 node-01 
    50      868.2gb       2tb      1.5tb      3.5tb           56 192.168.25.25 192.168.25.25 node-11 
    50      851.1gb       2tb      1.5tb      3.5tb           57 192.168.25.21 192.168.25.21 node-07 
    49      798.2gb       2tb      1.5tb      3.5tb           56 192.168.25.19 192.168.25.19 node-05 
    50        863gb       2tb      1.5tb      3.5tb           56 192.168.25.23 192.168.25.23 node-09 
    50      840.4gb       2tb      1.5tb      3.5tb           56 192.168.25.18 192.168.25.18 node-04 
    50      887.9gb       2tb      1.5tb      3.5tb           56 192.168.25.22 192.168.25.22 node-08 
    50      856.9gb       2tb      1.5tb      3.5tb           57 192.168.25.26 192.168.25.26 node-12 
    49      144.6gb     1.3tb      2.2tb      3.5tb           38 192.168.25.39 192.168.25.39 node-17 
    50       99.9gb     1.3tb      2.2tb      3.5tb           37 192.168.25.38 192.168.25.38 node-16 
    49      843.7gb       2tb      1.5tb      3.5tb           56 192.168.25.16 192.168.25.16 node-02 
    49      108.2gb     1.3tb      2.2tb      3.5tb           38 192.168.25.37 192.168.25.37 node-15 
    49      106.7gb     1.5tb        2tb      3.5tb           42 192.168.25.41 192.168.25.41 node-19 
    50      861.3gb     2.2tb      1.3tb      3.5tb           61 192.168.25.20 192.168.25.20 node-06 
    50      839.4gb       2tb      1.5tb      3.5tb           56 192.168.25.24 192.168.25.24 node-10 
    50       98.7gb     1.4tb      2.1tb      3.5tb           39 192.168.25.36 192.168.25.36 node-14 

# curl 127.0.0.1:9200/_cluster/health?pretty
{
  "cluster_name" : "mycluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 24,
  "number_of_data_nodes" : 24,
  "active_primary_shards" : 570,
  "active_shards" : 1140,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

# curl -s 127.0.0.1:9200/_cluster/settings?pretty
{
  "persistent" : {
    "indices" : {
      "recovery" : {
        "max_bytes_per_sec" : "200mb"
      }
    },
    "discovery" : {
      "zen" : {
        "minimum_master_nodes" : "21"
      }
    },
    "cluster" : {
      "routing" : {
        "rebalance" : {
          "enable" : "all"
        },
        "allocation" : {
          "enable" : "all"
        }
      }
    }
  },
  "transient" : {
    "indices" : {
      "recovery" : {
        "concurrent_streams" : "5"
      }
    },
    "cluster" : {
      "routing" : {
        "rebalance" : {
          "enable" : "all"
        },
        "allocation" : {
          "enable" : "all",
          "disable_replica_allocation" : "false",
          "cluster_concurrent_rebalance" : "10",
          "allow_rebalance" : "always",
          "disable_allocation" : "false"
        }
      }
    }
  }
}

[Christian_Dahlqvist]

Are all nodes on exactly the same version? How come it lists 1.8tb as used disk space?

[mobidyc]

Well done, all nodes are 2.3.4 except the problematic node which is 2.3.0.
the disk used space contains elasticsearch and other log datas

I will upgrade this node and check if it solves my problem

thank you

[mobidyc]

Update done, all my nodes are in version 2.3.4 now.

issue solved

thank you again