# Trying to take Indices Backup for full cluster restart **URL:** \ **Category:** Elasticsearch\ **Created:** [August 30, 2017, 11:55am UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851 "2017-08-30T11:55:22Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [August 30, 2017, 11:55am UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851/1 "2017-08-30T11:55:22Z")
Hi Trying to take backup of Prod indices using ``` curl -XPUT 'elk.***.com:9200/_snapshot/my_backup' \ { "type": "fs", "settings": { "location": "/elkprod_bak/elkback", "compress": true } } ``` But getting the error ``` error": { "root_cause": [ { "type": "process_cluster_event_timeout_exception", "reason": "failed to process cluster event (put_repository [my_backup]) within 30s" } ], "type": "process_cluster_event_timeout_exception", "reason": "failed to process cluster event (put_repository [my_backup]) within 30s" }, "status": 503 ``` ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [August 31, 2017, 8:36am UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851/2 "2017-08-31T08:36:06Z")
The returned message shows, that your request could not be processed within 30 seconds. Create a repository is not a heavy process, so I dont think this particular request is causing issues, but your cluster may be pretty busy. Can you check the pending tasks and the cluster health an paste it here? ```auto GET _cluster/health GET _cat/pending_tasks ``` ---
**Author:** ![harish.g](https://avatars.discourse-cdn.com/v4/letter/h/e495f1/32.png) [@harish.g](https://discuss.elastic.co/u/harish.g)\ **Post date:** [September 1, 2017, 6:28am UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851/3 "2017-09-01T06:28:01Z")
Cluster health : { "cluster\_name": "csm\_elk\_es\_whq\_1", "status": "red", "timed\_out": false, "number\_of\_nodes": 3, "number\_of\_data\_nodes": 3, "active\_primary\_shards": 2313, "active\_shards": 2469, "relocating\_shards": 0, "initializing\_shards": 8, "unassigned\_shards": 8195, "delayed\_unassigned\_shards": 0, "number\_of\_pending\_tasks": 236, "number\_of\_in\_flight\_fetch": 0, "task\_max\_waiting\_in\_queue\_millis": 353214, "active\_shards\_percent\_as\_number": 23.135307346326837 } Also cluster is not stable and pending tasks are keep on increasing ,which result ELK is going down completely We have clustered three nodes (00,01 and 02 nodes ) .If 02 become then backup will be successfully but its not backup complete data . "logstash-2015.11.11", ".kibana", "logstash-2015.11.12", "logstash-2015.11.10", "wsiasp-2017.03.30", "wsiasp-2017.03.31", "logstash-2015.11.01" ], "state": "FAILED", "reason": "Indices don't have primary shards [logstash-2017.07.02, wsiasp-2017.08.08]", "start\_time": "2017-09-01T07:31:33.325Z", "start\_time\_in\_millis": 1504251093325, "end\_time": "2017-09-01T07:31:42.748Z", "end\_time\_in\_millis": 1504251102748, "duration\_in\_millis": 9423, "failures": [ { "index": "logstash-2017.07.02", "shard\_id": 0, "reason": "primary shard is not allocated", "status": "INTERNAL\_SERVER\_ERROR" }, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 1, 2017, 9:23am UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851/4 "2017-09-01T09:23:40Z")
You seem to have far too many shards for a cluster that size, which most likely is why you are having problems with performance and stability. You need to either scale the cluster up/out or dramatically reduce the number of shards, e.g. by deleting data or reindexing into consolidated indices with fewer shards. ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [September 1, 2017, 9:37am UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851/5 "2017-09-01T09:37:30Z")
Thanks for the response. I have 3 nodes ES cluster which has around 1 TB of data and just 2 indices. So you are saying to either add 1 more node to cluster or have multiple indices so that there will be fewer shards per indices. Correct me if am wrong. Addition to this i have 4.4 Billion documents and 1100 indices with 2.1 TB data all the nodes put together. ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/f/0/f0d9e52d5d913b27d811d52f332af430660386c8.PNG) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 1, 2017, 10:06am UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851/6 "2017-09-01T10:06:02Z")
You have over 10000 shards, so if you are using the default 5 primary shards and 1 replica, that is over 1000 indices. If you have 2TB of data in the cluster, your average shard size is around 200MB, which is quite small. If you have a long retention period (looks like that based on the number of indices) I would suggest switching to monthly indices with 3-5 primary shards. That would give an average shard size roughly between 5GB and 10GB, which is quite common size. ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [September 4, 2017, 12:23pm UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851/7 "2017-09-04T12:23:52Z")
So how to go about taking snapshots of the indices? Do I need to reindex with new shards settings? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 4, 2017, 1:07pm UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851/8 "2017-09-04T13:07:02Z")
You may need to get the cluster into a better shape by reducing the number of shards before you take a snapshot. For ways to reduce the number of shards, have a look at the [reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs-reindex.html) and the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/indices-shrink-index.html). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 2, 2017, 1:07pm UTC](https://discuss.elastic.co/t/trying-to-take-indices-backup-for-full-cluster-restart/98851/9 "2017-10-02T13:07:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.