Shards UNASSIGNED NODE_LEFT

Elastic Stack Elasticsearch
1

Good day,
for some reason my shards are showing up in my cluster as UNASSIGNED NODE_LEFT. i wrote a script based on recommendation on the web. however it doe not relocate the shards... please help

[root@essep02 tmp]# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 39431 100 39431 0 0 291k 0 --:--:-- --:--:-- --:--:-- 291k
winlogbeat-2016.08.26 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.21 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.21 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.03 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.03 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.03 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.03 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.03 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.03 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.03 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.07.15 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.08.15 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.08.15 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.09.07 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.09.07 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.06 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.06 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.06 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.06 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.08.06 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.05.28 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.25 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.25 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.25 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.07.31 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.07.31 1 p UNASSIGNED NODE_LEFT

[root@essep03 ~]# for shard in $(curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print $2}'); do

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
    "commands" : [ {
          "allocate" : {
              "index" : "winlogbeat-*",
              "shard" : $shard,
              "node" : "essep035",
              "allow_primary" : true
          }
        }
    ]
}'
sleep 5

done

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 71804 100 71804 0 0 186k 0 --:--:-- --:--:-- --:--:-- 186k
{"error":{"root_cause":[{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}],"type":"parsing_exception","reason":"[cluster_reroute] failed to parse field [commands]","line":3,"col":28,"caused_by":{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}},"status":400}{"error":{"root_cause":[{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}],"type":"parsing_exception","reason":"[cluster_reroute] failed to parse field [commands]","line":3,"col":28,"caused_by":{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}},"status":400}{"error":{"root_cause":[{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}],"type":"parsing_exception","reason":"[cluster_reroute] failed to parse field [commands]","line":3,"col":28,"caused_by":{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}},"status":400}^C

2

Did a node leave?

That is probably not the best.

3

so what do I do to fix it, I have 273 in assigned shards

4

The node that had the shards that are unassigned now has crashed / was shut down and there is no other node with that data (as you didn't configure replicas). If you don't care about the data, feel free to delete the indices.

5

anything I can do to get it back?

6

start up the node that had the data that's now missing?

7

no [allocation_command] registered for [allocate]

It looks to me in 5.0 the cluster reroute command changed and no longer accepts the "allocate" parameter. That would explain this error.

From the documentation for cluster/reroute in previous versions of ES:

'allocate'
Allocate an unassigned shard to a node. Accepts the index and shard for index name and shard number, and node to allocate the shard to. It also accepts allow_primary flag to explicitly specify that it is allowed to explicitly allocate a primary shard (might result in data loss).

Do you see something else that indicates the data is missing altogether?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.