Shards UNASSIGNED NODE_LEFT


(Alvaro Cabrera) #1

Good day,
for some reason my shards are showing up in my cluster as UNASSIGNED NODE_LEFT. i wrote a script based on recommendation on the web. however it doe not relocate the shards... please help

[root@essep02 tmp]# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 39431 100 39431 0 0 291k 0 --:--:-- --:--:-- --:--:-- 291k
winlogbeat-2016.08.26 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.21 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.21 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.03 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.03 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.03 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.03 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.03 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.03 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.03 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.07.15 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.08.15 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.08.15 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.09.07 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.09.07 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.06 1 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.06 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.06 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.06.06 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.08.06 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.05.28 0 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.25 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.25 4 p UNASSIGNED NODE_LEFT
winlogbeat-2016.10.25 2 p UNASSIGNED NODE_LEFT
winlogbeat-2016.07.31 3 p UNASSIGNED NODE_LEFT
winlogbeat-2016.07.31 1 p UNASSIGNED NODE_LEFT


[root@essep03 ~]# for shard in $(curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print $2}'); do

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
    "commands" : [ {
          "allocate" : {
              "index" : "winlogbeat-*",
              "shard" : $shard,
              "node" : "essep035",
              "allow_primary" : true
          }
        }
    ]
}'
sleep 5

done


% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 71804 100 71804 0 0 186k 0 --:--:-- --:--:-- --:--:-- 186k
{"error":{"root_cause":[{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}],"type":"parsing_exception","reason":"[cluster_reroute] failed to parse field [commands]","line":3,"col":28,"caused_by":{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}},"status":400}{"error":{"root_cause":[{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}],"type":"parsing_exception","reason":"[cluster_reroute] failed to parse field [commands]","line":3,"col":28,"caused_by":{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}},"status":400}{"error":{"root_cause":[{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}],"type":"parsing_exception","reason":"[cluster_reroute] failed to parse field [commands]","line":3,"col":28,"caused_by":{"type":"parsing_exception","reason":"no [allocation_command] registered for [allocate]","line":3,"col":28}},"status":400}^C


(Mark Walkom) #2

Did a node leave?

That is probably not the best.


(Alvaro Cabrera) #3

so what do I do to fix it, I have 273 in assigned shards


(Yannick Welsch) #4

The node that had the shards that are unassigned now has crashed / was shut down and there is no other node with that data (as you didn't configure replicas). If you don't care about the data, feel free to delete the indices.


(Alvaro Cabrera) #5

anything I can do to get it back?


(Yannick Welsch) #6

start up the node that had the data that's now missing?


(Ben Davis) #7

no [allocation_command] registered for [allocate]

It looks to me in 5.0 the cluster reroute command changed and no longer accepts the "allocate" parameter. That would explain this error.

From the documentation for cluster/reroute in previous versions of ES:

'allocate'
Allocate an unassigned shard to a node. Accepts the index and shard for index name and shard number, and node to allocate the shard to. It also accepts allow_primary flag to explicitly specify that it is allowed to explicitly allocate a primary shard (might result in data loss).

Do you see something else that indicates the data is missing altogether?


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.