Continuing the discussion on why the shard movement happens though the design is correct to not to have movement and to pick local shards.
Original Issue that was closed till the discussion is over:
Ability to stop and start a cluster without shard movement
Shard Movement When each nodes had different number of shards
Log files from the ES master node has been attached here.
§ Output of below commands before shutting down ES nodes, can be found in the mentioned text files which are part of zipped file.
curl -XGET 'elasticsearch1:9200/_stats?level=shards' > before-shutdown-stats-shards.txt
curl -XGET 'elasticsearch1:9200/_shard_stores?status=green,yellow,red' > before-shutdown-shard_stores-status.txt
§ Output of below commands after reboot of ES nodes but before re-enabling allocation, can be found in the mentioned text files which are also part of zipped file.
curl -XGET 'elasticsearch1:9200/_stats?level=shards' > after-shutdown-stats-shards.txt
curl -XGET 'elasticsearch1:9200/_shard_stores?status=green,yellow,red' > after-shutdown-shard_stores-status.txt
Observation:
· There were no shard movement during the allocation.
[root@elasticsearch7 elasticsearch]# curl -XGET 'elasticsearch2:9200/_cluster/health?pretty'
{
"cluster_name" : "elastic-search-cluster",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 7,
"number_of_data_nodes" : 7,
"active_primary_shards" : 716,
"active_shards" : 1431,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 99.93016759776536
}
· However there were 2 shard reallocation after cluster was “green” and 100% active shards, to balance the number of shards per nodes.
[root@elasticsearch7 elasticsearch]# curl -XGET 'elasticsearch2:9200/_cluster/health?pretty'
{
"cluster_name" : "elastic-search-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 7,
"number_of_data_nodes" : 7,
"active_primary_shards" : 716,
"active_shards" : 1432,
"relocating_shards" : 2,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Shard Movement when each node had equal number of shards
-
We tested ES cluster stop and start again with the equal number of shards (202) in all the 7 nodes.
[root@elasticsearch5 elasticsearch]# curl 'elasticsearch3:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
202 57.5gb 65.1gb 82.3gb 147.5gb 44 10.0.0.26 10.0.0.26 idNEX8h
202 57gb 64.6gb 82.8gb 147.5gb 43 10.0.0.23 10.0.0.23 lURUCVH
202 57.6gb 65.2gb 82.2gb 147.5gb 44 10.0.0.27 10.0.0.27 wht1qYx
202 57.3gb 64.9gb 82.5gb 147.5gb 44 10.0.0.18 10.0.0.18 zlHb2gq
202 57.1gb 64.7gb 82.7gb 147.5gb 43 10.0.0.25 10.0.0.25 7sNEZsb
202 57.9gb 65.5gb 81.9gb 147.5gb 44 10.0.0.21 10.0.0.21 70eRJrE
202 57gb 64.6gb 82.8gb 147.5gb 43 10.0.0.17 10.0.0.17 Y0EJjkY
-
Below is the output after disabling the allocation but before shutting down nodes. Number of shards is not even in 7 ES nodes and also we can see 707 shards unassigned.
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
152 42.4gb 64.6gb 82.8gb 147.5gb 43 10.0.0.20 10.0.0.20 Y0EJjkY
119 45.7gb 65.5gb 81.9gb 147.5gb 44 10.0.0.17 10.0.0.17 70eRJrE
90 13.2gb 64.7gb 82.7gb 147.5gb 43 10.0.0.21 10.0.0.21 7sNEZsb
139 43.7gb 64.9gb 82.5gb 147.5gb 44 10.0.0.18 10.0.0.18 zlHb2gq
0 0b 65.2gb 82.2gb 147.5gb 44 10.0.0.23 10.0.0.23 wht1qYx
55 8gb 64.7gb 82.7gb 147.5gb 43 10.0.0.22 10.0.0.22 lURUCVH
152 47.7gb 65.1gb 82.3gb 147.5gb 44 10.0.0.19 10.0.0.19 idNEX8h
707 UNASSIGNED
[root@elasticsearch7 share]# curl -XGET 'elasticsearch2:9200/_cluster/health?pretty'
{
"cluster_name" : "elastic-search-cluster",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 7,
"number_of_data_nodes" : 7,
"active_primary_shards" : 707,
"active_shards" : 707,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 707,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}
-
Below is the output after enabling the allocation but after restarting the nodes. We can see the relocation of shards happening to balance the number of shards in the cluster nodes.
Even though the number of shards initially were even in all 7 nodes, relocation of shards happened after the cluster turning “green” which is not in our control.
[root@elasticsearch7 share]# curl -XGET 'elasticsearch2:9200/_cluster/health?pretty'
{
"cluster_name" : "elastic-search-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 7,
"number_of_data_nodes" : 7,
"active_primary_shards" : 707,
"active_shards" : 1414,
"relocating_shards" : 2,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Seems there is some confusion with developers in Github and here. I am not able to upload a Zip or tar file, so have uploaded in the original issue in github.