Shard Failed - Too many levels of symbolic links


(Gonzalo ) #1

Hello everybody!!

It´s my first post here. I´ve been looking for and answer to my problem but i didn´t find anything, so there it goes.
I have a cluster with three nodes whit version 5.1.1.
As you can see there and some of them STARTED or INITIALIZING:

data 2 p UNASSIGNED
data 2 r UNASSIGNED
data 4 p UNASSIGNED
data 4 r UNASSIGNED
data 1 p UNASSIGNED
data 1 r UNASSIGNED
data 3 r INITIALIZING 192.168.222.80 node2
data 3 p STARTED 47274837 18.8gb 192.168.222.90 node3
data 0 r INITIALIZING 192.168.222.80 node2
data 0 p STARTED 47242645 18.8gb 192.168.222.90 node3

So I tried to reallocate the UNASSIGNED to one of the node. Whit the error "Too many levels of symbolic links".
I don´t know what does it mean or how I could correct it. I don´t know even how I came to this situation.
Please HELP ME!!

The code I used to reallocate the primary shards was this one:

POST /_cluster/reroute
{

"commands": [

        {
      "allocate_replica" : {
            "index" : "data", "shard" : 2,
            "node" : "DqGrN2frR0iOWiLg0VBKcQ"
      }

}

]

Thanks =))


(Ed) #2

Where is this error showing up in, and how are you sending the command?

so you have the entire message as well as any other information around the log line.

But too many symbolic links generally means you made a FileSystem error where a symlink points back to itself


(Gonzalo ) #3

Hi eperry!

Thanks for your reply. I´m gonna try to answer all your questions the best I can:
Where is this error showing up in?
I can see this issue everytime I watch the log file of node2(called lnxcol2). This is an example of the error:
Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/lnxcol02/nodes/0/indices/Ti5V6c8ASZuzlbiFy6il0g/0/index: Too many levels of symbolic links
how are you sending the command?
I´m sending this command from the Dev Tools in Kibana but I guess that it can´t be completed due the error "Too many levels of symbolic links" so this is the error I'm currently focused in

I would like to know:
How elastic came to the error "Too many levels of symbolic links"?
Is there any way to check the symbolic links?
Would it be safe for the data to remove them?

Thanks a lot.


(Gonzalo ) #4

Update:

I´ve typed ls -la on all the directories from "var" to "index" but I´ve haven´t found any symbolic links.
I´ve three nodes joining a cluster and all of them have the same error if a type tail -F /var/log/elasticsearch/index1.log

Caused by: java.nio.file.FileSystemException: /var/lib/elasticsearch/lnxcol03/nodes/0/indices/Ti5V6c8ASZuzlbiFy6il0g/0/index: Too many levels of symbolic links
at sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]

Any ideas?

Thanks!!


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.