Elaticsearch not recognising path repo

vishu.kolki · September 21, 2017, 2:25pm

Hi,

My elasticsearch applications are running on docker. i have mounted the path.repo but i am getting following error:

{"error":"RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.repositories.RepositoryException: [my_backup] location [/mnt/es-snaps] doesn't match any of the locations specified by path.repo\n at org.elasticsearch.repositories.fs.FsRepository.(Unknown Source)\n while locating org.elasticsearch.repositories.fs.FsRepository\n while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: RepositoryException[[my_backup] location [/mnt/es-snaps] doesn't match any of the locations specified by path.repo]; ","status":500}

Can you please help me out.
Thanks in advance.

Thank you

dadoonet · September 21, 2017, 3:31pm

What do you have in your elasticsearch.yml file?

vishu.kolki · September 21, 2017, 3:49pm

Hi David,

path.repo: ["/mnt/es-snaps"]

Following is the output for

curl "localhost:9200/_nodes/settings?pretty"

{
  "cluster_name" : "elasticsearch",
  "nodes" : {
    "BKJL7TS_Q5KC3dm8Xep2UA" : {
      "name" : "es-node-2",
      "transport_address" : "inet[/10.0.1.21:9300]",
      "host" : "7459d736d6e1",
      "ip" : "10.0.1.21",
      "version" : "1.7.5",
      "build" : "00f95f4",
      "http_address" : "inet[/10.0.1.21:9200]",
      "settings" : {
        "path" : {
          "logs" : "/usr/share/elasticsearch/logs",
          "home" : "/usr/share/elasticsearch",
          "repo" : "/usr/share/elasticsearch/snapshots"
        },
        "cluster" : {
          "name" : "elasticsearch"
        },
        "node" : {
          "name" : "es-node-2"
        },
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : "es-node-1"
              }
            }
          }
        },
        "name" : "es-node-2",
        "client" : {
          "type" : "node"
        },
        "foreground" : "yes",
        "config" : {
          "ignore_system_properties" : "true"
        },
        "script" : {
          "inline" : "on",
          "update" : "on",
          "groovy" : "on"
        }
      }
    },
    "-Nb8FWW_RPWL07yQCcD6QQ" : {
      "name" : "es-node-3",
      "transport_address" : "inet[/10.0.1.22:9300]",
      "host" : "8b0eda835741",
      "ip" : "10.0.1.22",
      "version" : "1.7.5",
      "build" : "00f95f4",
      "http_address" : "inet[/10.0.1.22:9200]",
      "settings" : {
        "path" : {
          "logs" : "/usr/share/elasticsearch/logs",
          "home" : "/usr/share/elasticsearch",
          "repo" : "/usr/share/elasticsearch/snapshots"
        },
        "cluster" : {
          "name" : "elasticsearch"
        },
        "node" : {
          "name" : "es-node-3"
        },
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : "es-node-1"
              }
            }
          }
        },
        "name" : "es-node-3",
        "client" : {
          "type" : "node"
        },
        "foreground" : "yes",
        "config" : {
          "ignore_system_properties" : "true"
        },
        "script" : {
          "inline" : "on",
          "update" : "on",
          "groovy" : "on"
        }
      }
    },
    "n5LJhtymRo2DL5I-FpTsag" : {
      "name" : "es-node-1",
      "transport_address" : "inet[/10.0.1.20:9300]",
      "host" : "79c839e26e38",
      "ip" : "10.0.1.20",
      "version" : "1.7.5",
      "build" : "00f95f4",
      "http_address" : "inet[/10.0.1.20:9200]",
      "settings" : {
        "path" : {
          "logs" : "/usr/share/elasticsearch/logs",
          "home" : "/usr/share/elasticsearch",
          "repo" : "/usr/share/elasticsearch/snapshots"
        },
        "cluster" : {
          "name" : "elasticsearch"
        },
        "node" : {
          "name" : "es-node-1"
        },
        "discovery" : {
          "zen" : {
            "ping" : {
              "unicast" : {
                "hosts" : "es-node-2"
              }
            }
          }
        },
        "name" : "es-node-1",
        "client" : {
          "type" : "node"
        },
        "foreground" : "yes",
        "config" : {
          "ignore_system_properties" : "true"
        },
        "script" : {
          "inline" : "on",
          "update" : "on",
          "groovy" : "on"
        }
      }
    }
  }
}

[ec2-user@ip-10-5-3-235 ~]$ curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
  "type": "fs",
  "settings": {
    "location": "/mnt/es-snaps",
    "compress": true
  }
}'

vishu.kolki · September 22, 2017, 6:25am

Hi David,
I checked the permissions of the mount point, still i get the same error as above.
Can you please help me out here?

Thanks in advance

dadoonet · September 22, 2017, 7:43am

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

I edited your post.

According to your Node settings, path.repo is set to /usr/share/elasticsearch/snapshots in all your nodes.

Did you restart? If you did, then you may be did not modify the right config file?

vishu.kolki · September 22, 2017, 8:09am

Hi David,

Thanks for your help.

When i ran this curl:
<curl -XPUT 'http://localhost:9200/_snapshot/ssg_backup' -d '{
"type": "fs",
"settings": {
"location": "/usr/share/elasticsearch/snapshots",
"compress": true
}
}'/>

I did not get any acknowledge message, i found the above error itself. But the repository was created when i checked it.
What may be the issue?

Thanks

dadoonet · September 22, 2017, 8:53am

Use this icon to format:

Can you share the full error log please?

vishu.kolki · September 22, 2017, 9:00am

Hi David,

Thanks for your help.
Sorry for texting it wrong way.

When i ran this curl:
Preformatted textcurl -XPUT 'http://localhost:9200/_snapshot/ssg_backup' -d '{
"type": "fs",
"settings": {
"location": "/usr/share/elasticsearch/snapshots",
"compress": true
}
}'Preformatted text

Preformatted text{"error":"RepositoryException[[my_backup] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.repositories.RepositoryException: [my_backup] location [/mnt/es-snaps] doesn't match any of the locations specified by path.repo\n at org.elasticsearch.repositories.fs.FsRepository.(Unknown Source)\n while locating org.elasticsearch.repositories.fs.FsRepository\n while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: RepositoryException[[my_backup] location [/mnt/es-snaps] doesn't match any of the locations specified by path.repo]; ","status":500}Preformatted text

I did not get any acknowledge message, i found the above error itself. But the repository was created when i checked it.
What may be the issue?

Thanks

dadoonet · September 22, 2017, 9:39am

Still wrong formatting.

Please look at the preview window. It will shows you that it's not good before posting.
And edit your post if you see it's badly formatted.

Is the error you are seeing an old one? Or does it appear exactly when you send the PUT command?

I mean that may be there is no issue if the repo is created after all and that's only the old log you got before?

vishu.kolki · September 22, 2017, 9:50am

`Thankyou David,

I ran the same curl to create new repo,

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{

"type": "fs",
"settings": {
"location": "/usr/share/elasticsearch/snapshots",
"compress": true
}
}'

{"error":"RepositoryVerificationException[[my_backup] [3wU4GDPRSF-ki8XrvQf06g, 'RepositoryVerificationException[[my_backup] store location [/usr/share/elasticsearch/snapshots] is not accessible on the node [[es-node-2][3wU4GDPRSF-ki8XrvQf06g][8ae0627e0323][inet[/10.0.1.21:9300]]]]; nested: FileNotFoundException[/usr/share/elasticsearch/snapshots/tests-LLIqYYcIRumsCSOxhuFQ1Q-3wU4GDPRSF-ki8XrvQf06g (Read-only file system)]; '], [c2tNSJqSR6GeRgDj-U_8DA, 'RemoteTransportException[[es-node-1][inet[/10.0.1.20:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] a file written by master to the store [/usr/share/elasticsearch/snapshots] cannot be accessed on the node [[es-node-1][c2tNSJqSR6GeRgDj-U_8DA][6be39e1793be][inet[/10.0.1.20:9300]]]. This might indicate that the store [/usr/share/elasticsearch/snapshots] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; '], [4oS8jsaBSXiS2zo7hO14ng, 'RemoteTransportException[[es-node-3][inet[/10.0.1.22:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_backup] a file written by master to the store [/usr/share/elasticsearch/snapshots] cannot be accessed on the node [[es-node-3][4oS8jsaBSXiS2zo7hO14ng][69171a5e5472][inet[/10.0.1.22:9300]]]. This might indicate that the store [/usr/share/elasticsearch/snapshots] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]; ']]]","status":500}

But when i checked, my_backup has been created.

Thankyou. `

dadoonet · September 22, 2017, 10:22am

About formatting: OMG. Please, that's super annoying.

Just select the full lines you want to format, then click on </> button.
Or wrap them using

```
CODE
```

That's simple as that.

About the error, it means that /usr/share/elasticsearch/snapshots is not a shared drive between the 3 nodes.
It must be shared.

vishu.kolki · September 22, 2017, 10:29am

Sorry again for the formatting part.

Since the elasticsearch is running in Docker container. 
/usr/share/elasticsearch/snapshots is mounted to /mnt/es-snaps (which is a NFS volume, shared with other two nodes.)
When i take a snapshot, the snapshot is taken to /mnt/es-snaps.
NFS is shareable but it is not working.
Do i have to make any change in the container level? Can you let me know what changes to make?

Thankyou

dadoonet · September 22, 2017, 10:51am

You don't have to format all your text but just the code.

Please...

Here elasticsearch is telling you that /usr/share/elasticsearch/snapshots is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node.

And I can also see Read-only file system. Which might give a clue here.

vishu.kolki · September 22, 2017, 12:45pm

Hi David,

Since it is in the docker i need to check some more.
I have intiated snapshot, but i can see it on only 1 node (cluster of 3 nodes), Is it taking snapshot of Cluster or a single node?

Thank you

dadoonet · September 22, 2017, 1:04pm

I have intiated snapshot, but i can see it on only 1 node (cluster of 3 nodes)

I'm unsure of what you are saying here.
It does not make sense to me. A snapshot is a backup of indices wherever the shards were allocated on.

A snapshot is not related to a node or to the number of nodes.

May be clarify what you are actually seeing?

vishu.kolki · September 22, 2017, 2:58pm

I mean, if i want to take snapshot, do i have to take it on all 3 nodes? Or taking snapshot on 1 node is enough. Since it is a cluster.

dadoonet · September 22, 2017, 4:39pm

Only one request is enough. The request is performed on the whole cluster

vishu.kolki · September 22, 2017, 4:52pm

Thankyou David.

I am checking with the permission issue on docker, will get back to you if i face any problem.

Thanks a lot for your help.

vishu.kolki · September 23, 2017, 2:38pm

Hi David,

After completion of snapshot i see 282 shards following details (3 node cluster):

"    end_time_in_millis" : 1506176704543,
    "duration_in_millis" : 6892813,
    "failures" : [ ],
    "shards" : {
    "total" : 282,
    "failed" : 0,
    "successful" : 282
    }

But when i check on server i see 396 shards:
    curl -XGET 'localhost:9200/_stats?pretty'
    {
      "_shards" : {
        "total" : 396,
        "successful" : 396,
        "failed" : 0
      },

Has it taken the full backup?

system · October 21, 2017, 2:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Repository exception - doesn't match any of the locations specified by path.repo because this setting is empty Elasticsearch	6	2786	May 11, 2017
Path.repo is empty Elasticsearch	15	60926	July 5, 2017
ES INSISTS on trying to create repos in wrong place Elasticsearch	7	743	October 19, 2017
Not able to add repository Elasticsearch	3	996	July 5, 2017
Getting error when doing back up Elasticsearch	4	1480	July 5, 2017

Elaticsearch not recognising path repo

Related topics