# How to take snapshots in cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [December 16, 2016, 6:57pm UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315 "2016-12-16T18:57:07Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![animageofmine](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@animageofmine](https://discuss.elastic.co/u/animageofmine)\ **Post date:** [December 16, 2016, 6:57pm UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/1 "2016-12-16T18:57:07Z")
Have a simple and a stupid question, since I couldn't find a direct answer (except [here](https://discuss.elastic.co/t/backing-up-a-multi-node-cluster-with--snapshot/62291)) We have a cluster of 8 servers, all dockerized with data volume mounted on the host (/var/lib/elasticsearch) and want to set up snapshots. A couple of quick stupid questions: 1. Do we take snapshot from each node in the cluster? 2. If answer to 1 is yes, do we create a separate snapshot (different name) for each node? E.g. Following are the nodes: esnode1-5, esmaster1-3 Following is the query for snapshot from each node in the cluster, where \ could be data1\_12162016 or data2\_12162016 and so on. ``` curl -XPUT 'http://localhost:9200/_snapshot/elasticsearch/?wait_for_completion=true' -d '{ "ignore_unavailable": "true", "include_global_state": false }' ``` When we restore, run the following API / request: ``` curl -XPOST 'localhost:9200/_snapshot/elasticsearch//_restore' -d '{ "ignore_unavailable": "true", "include_global_state": false }' ``` If answer to question 1 is no, please let me know how snapshots are taken for a dockerized cluster along with an example, if possible. Let me know if you need more information and thank you in advance. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 17, 2016, 12:22am UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/2 "2016-12-17T00:22:27Z")
A snapshot it a cluster level action, ie it happens on every node that holds data for the snapshot, and that node is responsible for putting the data into the repo. ---
**Author:** ![animageofmine](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@animageofmine](https://discuss.elastic.co/u/animageofmine)\ **Post date:** [December 17, 2016, 4:09am UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/3 "2016-12-17T04:09:20Z")
Thanks. I take your reply as "you need to back up from each node in the cluster". When we restore the snapshot, how would each node know what to restore from the repository? For example, if index has 5 shards, each shard on one data node, how would each data node know what to restore from snapshot? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 17, 2016, 4:14am UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/4 "2016-12-17T04:14:05Z")
Yes, all nodes. A restore does the entire index, or snapshot. You can't do per shard. ---
**Author:** ![animageofmine](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@animageofmine](https://discuss.elastic.co/u/animageofmine)\ **Post date:** [December 17, 2016, 4:29am UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/5 "2016-12-17T04:29:31Z")
Sounds good, I will set this up and give it a shot, thanks for your help! Just a quick clarification: Should I use the same snapshot name (e.g. snapshot\_12162016) from all nodes or separate snapshot name from each node? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 17, 2016, 4:35am UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/6 "2016-12-17T04:35:05Z")
You cannot have a per node snapshot. ---
**Author:** ![animageofmine](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@animageofmine](https://discuss.elastic.co/u/animageofmine)\ **Post date:** [December 17, 2016, 6:11am UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/7 "2016-12-17T06:11:12Z")
Perfect, thank you for the clarification! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 17, 2016, 7:11am UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/8 "2016-12-17T07:11:23Z")
Creating a snapshot is as Mark points out a cluster-level operation, and the generated snapshots are written to a shared storage that all nodes need to have access to, e.g. a network mounted file system, S3 or HDFS. ---
**Author:** ![animageofmine](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@animageofmine](https://discuss.elastic.co/u/animageofmine)\ **Post date:** [December 19, 2016, 7:08pm UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/9 "2016-12-19T19:08:11Z")
@Christian_Dahlqvist @warkolm I added AWS plugin and was able to register the repository. However, when I try to take snapshot from each of the nodes in the cluster, I get concurrent snapshot exception. ``` { "error": { "root_cause": [ { "type": "concurrent_snapshot_execution_exception", "reason": "[essnapshots:12192016]a snapshot is already running" } ], "type": "concurrent_snapshot_execution_exception", "reason": "[essnapshots:12192016]a snapshot is already running" }, "status": 503 } ``` Following is my query that runs from each node in the cluster: ``` curl -XPUT 'http://localhost:9200/_snapshot/essnapshots/12192016?wait_for_completion=false' -d '{ "ignore_unavailable": "true", "include_global_state": false }' ``` This means, you can't use the same snapshot name from each node in the cluster unless you use different snapshot name from each node in the cluster. Or you don't have to fire snapshot API from all the nodes in the cluster. I don't know which one is valid and would like to avoid trial and error test. A verbose explanation with an example would be appreciated. We are trying to go live in production. Thank you. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 19, 2016, 7:19pm UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/10 "2016-12-19T19:19:53Z")
Snapshots are not created per node but for the whole cluster, which is why all nodes need access to the storage. ---
**Author:** ![animageofmine](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@animageofmine](https://discuss.elastic.co/u/animageofmine)\ **Post date:** [December 19, 2016, 8:24pm UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/11 "2016-12-19T20:24:23Z")
I think all nodes have access to the storage because the snapshot was successful. This would mean that, you only need to trigger snapshot from one of the nodes in the cluster and it will talk with other nodes, grep the data and push it to S3. Let me know if that's not the case. Thanks so much! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 19, 2016, 8:45pm UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/12 "2016-12-19T20:45:00Z")
It does not matter which node you trigger the snapshot from, so that is correct. ---
**Author:** ![animageofmine](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@animageofmine](https://discuss.elastic.co/u/animageofmine)\ **Post date:** [December 19, 2016, 9:04pm UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/13 "2016-12-19T21:04:47Z")
sounds good. Thank you so much! ---
**Author:** ![animageofmine](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@animageofmine](https://discuss.elastic.co/u/animageofmine)\ **Post date:** [December 20, 2016, 4:19pm UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/14 "2016-12-20T16:19:51Z")
@Christian_Dahlqvist @warkolm Been monitoring the logs and observed the following, which seem to be from AWS connection. I think the logs are probably because of some internal protocol to connect with AWS, but I still wanted to run through you guys, if that's expected. I am not sure why is it connecting to AWS so many times that it has to close the connection over and over. We did not request any snapshots during this time. Seems to log every minute based on the pattern (6:04:11, 6:05:11, etc...) ``` [2016-12-20T06:04:11,002][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:05:11,002][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:06:11,002][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:07:11,002][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:08:11,003][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:09:11,003][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:10:11,003][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:11:11,004][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:12:11,004][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:13:11,004][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:14:11,005][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS [2016-12-20T06:15:11,005][DEBUG][o.a.h.i.c.PoolingClientConnectionManager] Closing connections idle longer than 60 SECONDS ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 17, 2017, 4:20pm UTC](https://discuss.elastic.co/t/how-to-take-snapshots-in-cluster/69315/15 "2017-01-17T16:20:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.