# Mass allocate\_stale\_primary
**URL:** https://discuss.elastic.co/t/mass-allocate-stale-primary/162719
**Category:** Elasticsearch
**Created:** [January 2, 2019, 10:10pm UTC](https://discuss.elastic.co/t/mass-allocate-stale-primary/162719 "2019-01-02T22:10:44Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@amomchilov](https://discuss.elastic.co/u/amomchilov)
#### Post date: [January 2, 2019, 10:10pm UTC](https://discuss.elastic.co/t/mass-allocate-stale-primary/162719/1 "2019-01-02T22:10:45Z")
We have a pretty sizable Elasticsearch cluster (currently storing 1.2 PB) that has had some data loss due to some PEBCAK. This is for a logging/debugging usecase, so we can tolerate data loss, it's a minor inconvenience, but not a big deal.
I _could_ delete all indices with assigned shards, but I would rather just `allocate_stale_primary` or `allocate_empty_primary` en masse to try and save as much data as possible.
My issue is that both of these APIs require a `node` parameter, which I don't know how to specify. Is there a way to have the cluster automatically determine which node to relocate on?
Alternatively, is there a way to ask the cluster which node would be a good candidate for a shard of an index, which I can then use as the argument to the `node` parameter?
Is there a better way? Is there a "yeah, screw the unassigned shards, I'm okay with dataloss, just make it green again" button I can easily reach for?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 3, 2019, 8:59am UTC](https://discuss.elastic.co/t/mass-allocate-stale-primary/162719/2 "2019-01-03T08:59:10Z")
> [@amomchilov](#):
>
> Is there a "yeah, screw the unassigned shards, I'm okay with dataloss, just make it green again" button I can easily reach for?
There is not, and that is deliberate. You will need to implement this yourself, and then ideally throw it away once done to avoid the temptation to use it again 🙂
For `allocate_empty_primary` it doesn't really matter which node to use, since you're allocating an empty primary. If there's a lot, probably best to spread them out a bit, but if you don't then they will be rebalanced eventually.
For `allocate_stale_primary` you need to find a stale copy of the shard first. `GET _shard_stores` or the [cluster allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) can do this. If you get it wrong and try and allocate a stale primary on a node that doesn't have a copy of the shard then, in 6.5.1 at least, it appears to succeed (returning `200 OK`) but in fact it fails a bit later on in the process. I opened [#37098](https://github.com/elastic/elasticsearch/issues/37098) to investigate this further.
---
### Author:  [@amomchilov](https://discuss.elastic.co/u/amomchilov)
#### Post date: [January 3, 2019, 8:26pm UTC](https://discuss.elastic.co/t/mass-allocate-stale-primary/162719/3 "2019-01-03T20:26:34Z")
> [@DavidTurner](#):
>
> There is not, and that is deliberate. You will need to implement this yourself, and then ideally throw it away once done to avoid the temptation to use it again
Unfortunately, our servers are managed by an internal system that's really aggressive about bringing down "failed" hosts (using a very loose definition of "failed"). We're at 1 replica per primary right now, which is almost good enough to compensate and consistently recover from failed servers, but not quite. Our use-case isn't important enough to cost justify jumping to 2 replicas per primary.
> [@DavidTurner](#):
>
> For `allocate_empty_primary` it doesn't really matter which node to use, since you're allocating an empty primary. If there's a lot, probably best to spread them out a bit, but if you don't then they will be rebalanced eventually.
It's like 800 shards. Is that still okay?
> [@DavidTurner](#):
>
> For `allocate_stale_primary` you need to find a stale copy of the shard first. `GET _shard_stores` or the [cluster allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) can do this. If you get it wrong and try and allocate a stale primary on a node that doesn't have a copy of the shard then, in 6.5.1 at least, it appears to succeed (returning `200 OK` ) but in fact it fails a bit later on in the process. I opened [#37098](https://github.com/elastic/elasticsearch/issues/37098) to investigate this further.
Thanks for investigating this!
So I essentially need to do a join. Use the rest API to fetch all unallocated shards (already done), and join it against the `_shard_stores` rest API to get the correct nodes, and use the result to call the reroute API. Is that correct?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 3, 2019, 9:51pm UTC](https://discuss.elastic.co/t/mass-allocate-stale-primary/162719/4 "2019-01-03T21:51:51Z")
> [@amomchilov](#):
>
> It's like 800 shards. Is that still okay?
Losing 800 shards' worth of data is stretching the bounds of what I'd call "okay", but it should be possible to allocate them all as empties if that's what you want. Spread them out so there's not too much rebalancing to do afterwards.
> [@amomchilov](#):
>
> Is that correct?
Sounds about right, yes. I think I'd do this first and only allocate the empty ones after everything has settled down and you've confirmed there's no more stale ones to go. If you allocate an empty primary then it'll wipe out any existing stale stores.
---
### Author:  [@amomchilov](https://discuss.elastic.co/u/amomchilov)
#### Post date: [January 4, 2019, 6:39pm UTC](https://discuss.elastic.co/t/mass-allocate-stale-primary/162719/5 "2019-01-04T18:39:48Z")
> [@DavidTurner](#):
>
> Losing 800 shards' worth of data is stretching the bounds of what I'd call "okay"
Heh, it's petabyte scale cluster. That's only about 2% of the shards by count, even less by actual data size
> [@DavidTurner](#):
>
> Spread them out so there's not too much rebalancing to do afterwards.
That's the hard part, IDK how to do that in a decent way 😕
> [@DavidTurner](#):
>
> I think I'd do this first and only allocate the empty ones after everything has settled down and you've confirmed there's no more stale ones to go. If you allocate an empty primary then it'll wipe out any existing stale stores.
Yep, that's exactly what I was thinking. Excellent. Thanks!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 1, 2019, 6:40pm UTC](https://discuss.elastic.co/t/mass-allocate-stale-primary/162719/6 "2019-02-01T18:40:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.