# Snapshot failed, all data cannot be restored

**URL:** https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286
**Category:** Elasticsearch
**Created:** [December 4, 2018, 4:42am UTC](https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286 "2018-12-04T04:42:48Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![Richardms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richardms/32/38401_2.png) [@Richardms](https://discuss.elastic.co/u/Richardms)
#### Post date: [December 4, 2018, 4:42am UTC](https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286/1 "2018-12-04T04:42:48Z")

</div>

I have 3 nodes, lets say node1, node1, and node3

i have succesfully restored the data from my nodes,  
but the data decrease fro, 380k data to 77k data.

this is what i get from backup data  
"hits": {  
"total": 76908,  
"max\_score": 1,

and this is my data  
"hits": {  
"total": 384959,  
"max\_score": 1,

how can i get all my data?

i want to migrate my ES Data to another VM / IP / Instances

---

<div class="post-metadata">

### Author: ![mjunaidmuzammil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mjunaidmuzammil/32/56910_2.png) [@mjunaidmuzammil](https://discuss.elastic.co/u/mjunaidmuzammil)
#### Post date: [December 4, 2018, 5:56am UTC](https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286/2 "2018-12-04T05:56:27Z")

</div>

Can you elaborate more about it?

1. How you performed snapshot and restore operations?
2. Did you check the index document size at the time of snapshot?
3. Did the snapshot complete successfully?

---

<div class="post-metadata">

### Author: ![Richardms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richardms/32/38401_2.png) [@Richardms](https://discuss.elastic.co/u/Richardms)
#### Post date: [December 4, 2018, 6:31am UTC](https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286/3 "2018-12-04T06:31:51Z")

</div>

this is exactly step i used to restore my data :

1. PUT /\_snapshot/es\_backup?verify=false  
{  
"type": "fs",  
"settings": {  
"location": "/etc/elasticsearch/es\_backup",  
"compress": true  
}  
}

* * *

1. PUT /\_snapshot/es\_backup/4dec1048  
{  
"indices": "log",  
"ignore\_unavailable": true,  
"include\_global\_state": false  
}

* * *

1. and then i try to see my data , so i restored it on the same nodes with :  
POST \_snapshot/es\_backup/4dec1048/\_restore  
{  
"indices": "log",  
"rename\_pattern": "log"  
,"rename\_replacement": "log-backup"  
}
2. then i want to see how much document that i have :  
GET log-backup/log/\_search

my data that i want to back up is 222.mb with 380k document  
and the data that i backuped 44,7mb with 77k document

es\_backup status :  
"snapshots": [  
{  
"snapshot": "4dec1048",  
"repository": "es\_backup",  
"uuid": "5KphFPQVQ0Sp6o92KvOjfw",  
"state": "SUCCESS",  
"include\_global\_state": false,  
"shards\_stats": {  
"initializing": 0,  
"started": 0,  
"finalizing": 0,  
"done": 0,  
"failed": 0,  
"total": 0  
},  
"stats": {  
"number\_of\_files": 0,  
"processed\_files": 0,  
"total\_size\_in\_bytes": 0,  
"processed\_size\_in\_bytes": 0,  
"start\_time\_in\_millis": 0,  
"time\_in\_millis": 0  
},  
"indices": {}  
}  
]

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 4, 2018, 7:48am UTC](https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286/4 "2018-12-04T07:48:30Z")

</div>

Thank you for sharing the exact commands you're using, it makes it so much easier to help.

I see a couple of strange things:

> [@Richardms](#):
>
> PUT /\_snapshot/es\_backup?verify=false

Why `?verify=false`?

> [@Richardms](#):
>
> "ignore\_unavailable": true,

Why not `false`?

These options are both asking Elasticsearch to be lenient in the case of problems, but it sounds like you do not want this lenience.

---

<div class="post-metadata">

### Author: ![Richardms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richardms/32/38401_2.png) [@Richardms](https://discuss.elastic.co/u/Richardms)
#### Post date: [December 4, 2018, 8:26am UTC](https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286/5 "2018-12-04T08:26:31Z")

</div>

Thankyou for your answer David!

This is why i turn verify to false :  
"reason": "[es\_logbak] [[uw5kPLo6TwC\_ZhKTlQat8Q, 'RemoteTransportException[[node-05][10.32.12.85:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[es\_logbak] a file written by master to the store [/etc/elasticsearch/es\_backup] cannot be accessed on the node [{node-05}{uw5kPLo6TwC\_ZhKTlQat8Q}{LM5ec5DrTHWHxiGpaZaKpg}{10.32.12.85}{10.32.12.85:9300}]. This might indicate that the store [/etc/elasticsearch/es\_backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];'], [b1q4PAqMQseSROz3Xg7u0A, 'RemoteTransportException[[node-06][10.32.12.86:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[es\_logbak] a file written by master to the store [/etc/elasticsearch/es\_backup] cannot be accessed on the node [{node-06}{b1q4PAqMQseSROz3Xg7u0A}{ZHXd0YmNSEWx6ZK3wcQNuA}{10.32.12.86}{10.32.12.86:9300}]. This might indicate that the store [/etc/elasticsearch/es\_backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]"

i tried  
// "ignore\_unavailable": true,

but still same ☹

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 4, 2018, 8:36am UTC](https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286/6 "2018-12-04T08:36:44Z")

</div>

That'd explain it:

> [@Richardms](#):
>
> a file written by master to the store [/etc/elasticsearch/es\_backup] cannot be accessed on the node [{node-06}{b1q4PAqMQseSROz3Xg7u0A}{ZHXd0YmNSEWx6ZK3wcQNuA}{10.32.12.86}{10.32.12.86:9300}]. This might indicate that the store [/etc/elasticsearch/es\_backup] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node]

Your shared-filesystem repository (NFS?) is not properly accessible across all nodes in your cluster.

---

<div class="post-metadata">

### Author: ![Richardms](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richardms/32/38401_2.png) [@Richardms](https://discuss.elastic.co/u/Richardms)
#### Post date: [December 4, 2018, 10:09am UTC](https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286/7 "2018-12-04T10:09:35Z")

</div>

> Your shared-filesystem repository (NFS?) is not properly accessible across all nodes in your cluster.

ah i see, i try to fix this problem first.  
Thankyou David!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 1, 2019, 10:09am UTC](https://discuss.elastic.co/t/snapshot-failed-all-data-cannot-be-restored/159286/8 "2019-01-01T10:09:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
