# Issues with Backup repository setup & shards allocation

**URL:** <https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848>\
**Category:** Elasticsearch\
**Created:** [February 18, 2019, 2:18pm UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848 "2019-02-18T14:18:14Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\
**Post date:** [February 18, 2019, 2:18pm UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/1 "2019-02-18T14:18:14Z")

</div>

HI Team,  
Below is my production setup.

```
10.XX.YY.ZZ 25 99 13 1.81 1.52 1.29 md - prodcution_master_data_Server3

10.XX.YY.ZZ 41 99 7 0.25 0.25 0.29 d - prodcution_data_Server1

10.XX.YY.ZZ 21 99 10 1.61 1.47 1.28 - - prodcution_client_Server3

10.XX.YY.ZZ 6 99 4 0.25 0.25 0.29 m - prodcution_master_Server1

10.XX.YY.ZZ 72 99 16 0.83 0.77 0.93 md * prodcution_master_data_Server2

```

I am now setting up archival strategy here to take snapshot of all indices including the cluster state, but i am getting frequent shard allocation errors.

The default no of shards allocated is 5 and i have reduced the no of replica's to 1.

Also after setting up the backup repository, i did take few backup's for a cluster of indices but at on point i am getting a message "concurrent\_snapshot\_execution\_exception" which i killing me in proceeding further. I triedto query the status,but still couldn't get any information except snapshot

Is there anything which i could do here which would help me to proceed further and resolve all these problems, I am planning to reduce the no of shards to 3(will i be able to achieve it) and i will also setup cronjobs and curator soon here.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 18, 2019, 2:27pm UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/2 "2019-02-18T14:27:52Z")

</div>

> [@Nithani25](#):
>
> i am getting frequent shard allocation errors.

It's impossible to help without seeing the actual errors that you're seeing.

> [@Nithani25](#):
>
> i am getting a message "concurrent\_snapshot\_execution\_exception"

Similarly, it's impossible to help without seeing the full response. There will be a lot more detail than just this message.

---

<div class="post-metadata">

**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\
**Post date:** [February 25, 2019, 6:38am UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/3 "2019-02-25T06:38:06Z")

</div>

Ok, Here is my problem,  
I was able to take a backup(snapshot) of set of indices which i have indexed in the month of January with the below api.

```
PUT /_snapshot/index_backups/snapshot_jan19_index-name?wait_for_completion=true
{
  "indices": "index-name-2019.01.0*",
  "ignore_unavailable": true,
  "include_global_state": true
}

```

The snapshot activity was successfull, and have recieved below response.

`snapshot_jan19_index-name SUCCESS 1550560280 23:11:20 1550560301 23:11:41 21.3s 10 50 0 50`

Since indexing was successful, i tried deleting one index manually, so i could test restore operation

```
DELETE index-name-2019.01.11

```

I tried restoring the index and was successful again

```
POST /_snapshot/kindex_backups/snapshot_jan19_index-name/_restore
{
  "indices": "index-name-2019.01.11"
}

```

Restore is also successful, but unable to get the status of the restore index yet,instead getting the below error message.

```
GET /index-name-2019.01.11/_count

{
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": []
  },
  "status": 503
}
```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 25, 2019, 8:29am UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/4 "2019-02-25T08:29:46Z")

</div>

Thanks, that is very helpful.

Restores are, by default, asynchronous operations: "success" means the cluster has successfully started to restore the snapshot, not that it has completed. If you want to wait for the restore to complete, set the `wait_for_completion` parameter:

```nohighlight
POST /_snapshot/kindex_backups/snapshot_jan19_index-name/_restore?wait_for_completion=true
{
  "indices": "index-name-2019.01.11"
}

```

Additionally you can monitor the progress of the restore operation using the [indices recovery API](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/indices-recovery.html), or by waiting for all the shards to be allocated using the [cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html):

```nohighlight
GET /_cluster/health?wait_for_status=green

```

---

<div class="post-metadata">

**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\
**Post date:** [February 26, 2019, 5:51am UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/5 "2019-02-26T05:51:49Z")

</div>

Hi Turner,  
Thanks for the information, I understand that the snapshot can only be restored when the cluster status is green. But i have a problem here, my cluster status is showing up Red for a long time and snapshots are little baby steps which i have taken to remove old and unwanted indices so i can save some space in the allocated servers. Also i see this unassigned shards coming up more frequently which i believe is the cause for the cluster being Red.

```
GET /_cluster/health

{
  "cluster_name": "production_cluster",
  "status": "red",
  "timed_out": false,
  "number_of_nodes": 5,
  "number_of_data_nodes": 3,
  "active_primary_shards": 5349,
  "active_shards": 10520,
  "relocating_shards": 0,
  "initializing_shards": 0,
  _ **"unassigned_shards": 189,** _
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 98.23512933046969
}
```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 26, 2019, 8:23am UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/6 "2019-02-26T08:23:29Z")

</div>

You have over 10,000 active shards in this 3-node cluster. **This is far too many and will cause you problems.** Here is an article that gives more detail:

> **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**

> [@Nithani25](#):
>
> I understand that the snapshot can only be restored when the cluster status is green.

I don't think this is the case. It should be possible to restore a snapshot whatever your cluster health (as long as it has a master, of course).

> [@Nithani25](#):
>
> i see this unassigned shards coming up more frequently which i believe is the cause for the cluster being Red.

Indeed, a health of `red` means there are unassigned primary shards. To find the shards that are unassigned you can use `GET _cat/shards`, and then you can investigate why they are unassigned using the [allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html).

---

<div class="post-metadata">

**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\
**Post date:** [February 27, 2019, 7:22am UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/7 "2019-02-27T07:22:02Z")

</div>

Hi Turner,  
Thanks for pointing the problem of accumulating indexes within 3 node production cluster, i am tryning to look into the possible resolution steps for the same.

I am planning to reduce the no of shards to be 3 for all indexes and will only one replica.

But my current status of the cluster seems to red because of these many shards, Considering this what would be the best option to take this forward

shrink indexes or reindexing!!

Also,since i have snapshot backup of all january indices can i go ahead and delete them to reduce the no of shards!! -- I am asking this beacuse i haven't tested the restore activity yet.

And finally can i move the contents of snapshot repository to another backup server completely.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 27, 2019, 8:28am UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/8 "2019-02-27T08:28:46Z")

</div>

> [@Nithani25](#):
>
> what would be the best option to take this forward

It depends on why your cluster is currently red, hence...

> [@Nithani25](#):
>
> But my current status of the cluster seems to red because of these many shards

... can you explain the reasoning behind that conclusion?

> [@Nithani25](#):
>
> I am planning to reduce the no of shards to be 3 for all indexes and will only one replica.

Why not 1 shard? How big is each index? Are these daily indices? If so, can you move to weekly or monthly ones instead?

> [@Nithani25](#):
>
> since i have snapshot backup of all january indices can i go ahead and delete them to reduce the no of shards!! -- I am asking this beacuse i haven't tested the restore activity yet.

Probably wise to test that restores work first. You can also just [close](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-open-close.html) some unneeded indices for now while you get your cluster under control.

---

<div class="post-metadata">

**Author:** ![Nithani25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nithani25/32/33457_2.png) [@Nithani25](https://discuss.elastic.co/u/Nithani25)\
**Post date:** [March 13, 2019, 6:42am UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/9 "2019-03-13T06:42:10Z")

</div>

Hi David,  
After long struggle, i can only reach status where my cluster status has turned yellow and still there too many open shards available, I will consider your valuable suggestion of changing all daily indices into weekly ones, while i start doing it

how can i change their defaults shard allocation to be 1 along with replica as 1!!

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 13, 2019, 9:12am UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/10 "2019-03-13T09:12:16Z")

</div>

> [@Nithani25](#):
>
> i can only reach status where my cluster status has turned yellow and still there too many open shards available

It's really hard to help here because you're not sharing much information and are possibly jumping to conclusions. A yellow health means there are unassigned shards, but this isn't normally because there's too many open shards. As I said above:

> [@DavidTurner](#):
>
> To find the shards that are unassigned you can use `GET _cat/shards`, and then you can investigate why they are unassigned using the [allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html).

> [@Nithani25](#):
>
> how can i change their defaults shard allocation to be 1 along with replica as 1!!

The number of shards comes from the index settings which probably come from the index template. To adjust the default number of shards, change the index template.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 10, 2019, 9:12am UTC](https://discuss.elastic.co/t/issues-with-backup-repository-setup-shards-allocation/168848/11 "2019-04-10T09:12:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.