Snapshot Failure

[Gnosis]

Hi all,

I'd like to make a snapshot of one index from my three node cluster.
Each node is on a different computer.
Each node has the same conf and the same path.repo.
Each path to the repo is 777 (i know it's bad, one thing at a time) .

I've created my repo on the master node with :

curl -XPUT 'http://localhost:9200/_snapshot/bckup?pretty' -d '{
"type": "fs",
"settings": {
"location": "/pfic/elasticsearch/repo/bckup",
"compress": true
}
}'

Here i have the error :

{
"error" : {
"root_cause" : [ {
"type" : "repository_exception",
"reason" : "[bckup] failed to create repository"
} ],
"type" : "repository_exception",
"reason" : "[bckup] failed to create repository",
"caused_by" : {
"type" : "creation_exception",
"reason" : "Guice creation errors:\n\n1) Error injecting constructor, java.nio.file.FileSystemException: /pfic/elasticsearch/repo/bckup: No space left on device\n at org.elasticsearch.repositories.fs.FsRepository.(Unknown Source)\n while locating org.elasticsearch.repositories.fs.FsRepository\n while locating org.elasticsearch.repositories.Repository\n\n1 error",
"caused_by" : {
"type" : "file_system_exception",
"reason" : "/pfic/elasticsearch/repo/bckup: No space left on device"
}
}
},
"status" : 500
}

And i've tried to snapshot an index

curl -XPUT 'http://localhost:9200/_snapshot/bckup/snapshot2?wait_for_completion=true
{
"indices": "idx_test",
"ignore_unavailable": "true",
"include_global_state": false
}'

Here i have the error :

"indices": "idx_mys_bad_pmmqrmys-2016.04.29" -->
--curl--http://localhost:9200/_snapshot/bckup/snapshot2?pretty&wait_for_completion=true

"indices": "idx_mys_bad_pmmqrmys-2016.04.29"
{
"error" : {
"root_cause" : [ {
"type" : "repository_missing_exception",
"reason" : "[bckup] missing"
} ],
"type" : "repository_missing_exception",
"reason" : "[bckup] missing"
},
"status" : 404
}

"ignore_unavailable": "true" -->
--curl--http://localhost:9200/_snapshot/bckup/snapshot2?pretty&wait_for_completion=true

"ignore_unavailable": "true"
{
"error" : {
"root_cause" : [ {
"type" : "repository_missing_exception",
"reason" : "[bckup] missing"
} ],
"type" : "repository_missing_exception",
"reason" : "[bckup] missing"
},
"status" : 404
}

"include_global_state": false
-->
--curl--http://localhost:9200/_snapshot/bckup/snapshot2?pretty&wait_for_completion=true

"include_global_state": false

{
"error" : {
"root_cause" : [ {
"type" : "repository_missing_exception",
"reason" : "[bckup] missing"
} ],
"type" : "repository_missing_exception",
"reason" : "[bckup] missing"
},
"status" : 404
}

Someone could help me ?

Do i have to make the repo on each node ? something else ?

Thx all.

[theuntergeek (Aaron Mildenstein)]

The repo must be a shared filesystem that is accessible by all master and data nodes. That generally implies NFS in the Linux world.

It is not enough to have similar filesystem paths on separate machines. This is because shards and segments move around, and at snapshot time, any given node must be able to ensure that a segment is not being replicated needlessly, hence the need for a shared filesystem.