], but all the topics point to errors that don't seem to match my use-case. I've also been trying to understand these limits, as I had 8GB allocated to these VMs, but expanded to 16GB and the limit remains at 940.3mb. Should I change this to match my environment more realistically or is there a different issue impacting me here? It's also interesting to me that I see these same errors in logstash.log, which I thought would have nothing to do with the indexing portion.
W/o ?pretty (networking data redacted):
```
{
"index" : "private_index",
"shard" : 0,
"primary" : true,
"current_state" : "started",
"current_node" : {
"id" : "ihTj7kz6QQewpWM-FMo6CA",
"name" : "node1",
"transport_address" : "node1_ip:9300",
"attributes" : {
"ml.machine_memory" : "16656506880",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"weight_ranking" : 1
},
"can_remain_on_current_node" : "yes",
"can_rebalance_cluster" : "no",
"can_rebalance_cluster_decisions" : [
{
"decider" : "rebalance_only_when_active",
"decision" : "NO",
"explanation" : "rebalancing is not allowed until all replicas in the cluster are active"
},
{
"decider" : "cluster_rebalance",
"decision" : "NO",
"explanation" : "the cluster has unassigned shards and cluster setting [cluster.routing.allocation.allow_rebalance] is set to [indices_all_active]"
}
],
"can_rebalance_to_other_node" : "no",
"rebalance_explanation" : "rebalancing is not allowed",
"node_allocation_decisions" : [
{
"node_id" : "7LIgwS6NQEuF0H8S-lLZCA",
"node_name" : "node2",
"transport_address" : "node2_ip:9300",
"node_attributes" : {
"ml.machine_memory" : "16656506880",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[private_index][0], node[7LIgwS6NQEuF0H8S-lLZCA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=iOPv9-x_R26AMQQxMbyF_A], unassigned_info[[reason=ALLOCATION_FAILED], at[2019-07-10T16:39:15.841Z], failed_attempts[1], delayed=false, details[failed shard on node [7LIgwS6NQEuF0H8S-lLZCA]: master {node1}{ihTj7kz6QQewpWM-FMo6CA}{O6ARr-ijQSuaCDIYQ6phFQ}{node1_ip}{node1_ip:9300}{ml.machine_memory=16656506880, ml.max_open_jobs=20, xpack.installed=true} has not removed previously failed shard. resending shard failure], allocation_status[no_attempt]]]"
}
]
}
]
}
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 11, 2019, 2:27pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/2 "2019-07-11T14:27:45Z")
> [@penguinairlines](#):
>
> I apologize for the large number of edits, I had some trouble using the code blocks since I didn't perceive that an extra carriage return needed to precede the preformatted text.
No worries, thanks for putting the effort in to get it right. It makes it so much easier to help.
Re. issue 2:
> [@penguinairlines](#):
>
> I had 8GB allocated to these VMs, but expanded to 16GB and the limit remains at 940.3mb.
Have you adjusted the [amount of memory allocated to the heap of the JVM running Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/heap-size.html) to match? Although you've allocated more memory to your VM, it looks like you're still running Elasticsearch with the default 1GB heap size.
Re. issue 1, the allocation explain output is telling us that this (primary) shard is healthy (`"current_state" : "started"`) so I'm not sure what the problem there is. What was the command you used to get that output? Did you perhaps say `"primary": true` when you meant `"primary": false`?
---
### Author:  [@penguinairlines](https://discuss.elastic.co/u/penguinairlines)
#### Post date: [July 11, 2019, 2:55pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/3 "2019-07-11T14:55:07Z")
> [@DavidTurner](#):
>
> Re. issue 1, the allocation explain output is telling us that this (primary) shard is healthy ( `"current_state" : "started"` ) so I'm not sure what the problem there is. What was the command you used to get that output? Did you perhaps say `"primary": true` when you meant `"primary": false` ?
To be clear, the primary shard is on node1 and working just fine. I'm trying to initiate the replica shard, which is giving me all of these errors. is `"primary": false` the incorrect syntax?
I will look into growing the heap size now. I was under the impression that this was set to some percentage by default though.
edit\> `GET /_cluster/stats` shows:
```
Node1:
"mem" : {
"heap_used_in_bytes" : 456437880,
"heap_max_in_bytes" : 1037959168
Node2:
"mem" : {
"heap_used_in_bytes" : 1574579536,
"heap_max_in_bytes" : 2075918336
```
So I believe this is not yet a heap size issue. Am I reading this correctly? Also wondering why my max heap size is different per node... Is this the right command used to check this setting? I see an older suggestion to use `/_cluster/nodes/stats?jvm=true&pretty=true'` but this fails due to unrecognized parameters: [jvm], [nodeId]. The doc provided provides how to set the heap size but not how to check it, which I would like to do first to validate the issue.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 11, 2019, 3:24pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/4 "2019-07-11T15:24:14Z")
> [@penguinairlines](#):
>
> I was under the impression that this was set to some percentage by default though.
No, as per the docs I shared above, it defaults to 1GB and it's up to you to change this if you need it.
> [@penguinairlines](#):
>
> So I believe this is not yet a heap size issue.
Do you expect your nodes to have respectively a 1GB and a 2GB maximum heap size? It's odd that they're different indeed, and also that's not a lot of heap for a production cluster. The `circuit_breaking_exception` message you quoted indicates that the heap on the node with the 1GB heap is 95% full and this is the reason it's refusing to do any meaningful work.
> [@penguinairlines](#):
>
> To be clear, the primary shard is on node1 and working just fine. I'm trying to initiate the replica shard, which is giving me all of these errors. is `"primary": false` the incorrect syntax?
I'm not sure I follow. You shared the allocation explain output showing us that the primary is healthy. This is nice, of course, but I think you meant to share the output about the replica. For this, you should say `"primary": false` in your request to the allocation explain API.
---
### Author:  [@penguinairlines](https://discuss.elastic.co/u/penguinairlines)
#### Post date: [July 11, 2019, 3:38pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/5 "2019-07-11T15:38:56Z")
`GET /_nodes/stats` Returns some better data:
```
(Node1)
"jvm" : {
"timestamp" : 1562859040471,
"uptime_in_millis" : 88730628,
"mem" : {
"heap_used_in_bytes" : 764154968,
"heap_used_percent" : 73,
"heap_committed_in_bytes" : 1037959168,
"heap_max_in_bytes" : 1037959168,
"non_heap_used_in_bytes" : 144271624,
"non_heap_committed_in_bytes" : 188936192,
(Node2)
"jvm" : {
"timestamp" : 1562859040623,
"uptime_in_millis" : 88154937,
"mem" : {
"heap_used_in_bytes" : 984559200,
"heap_used_percent" : 94,
"heap_committed_in_bytes" : 1037959168,
"heap_max_in_bytes" : 1037959168,
"non_heap_used_in_bytes" : 144338712,
"non_heap_committed_in_bytes" : 182394880,
```
Indeed that percentage is too high, and I acknowledge what you say about the 1GB default. It's possible that during my research I found this % based on an older version. Either way, this is definitely an issue that needs to be addressed - I will report back once I grow the heap size.
edit \> `"heap_max_in_bytes" : 4225236992,` (on both nodes) looks like the size has grown successfully. Replica shard is still showing the same INITIALIZING error, so I'll repeat those error collection steps.
```
GET /_cluster/allocation/explain
{
"index": "private_index",
"shard": 0,
"primary": true
}
```
returns:
```
{
"index" : "private_index",
"shard" : 0,
"primary" : true,
"current_state" : "started",
"current_node" : {
"id" : "ihTj7kz6QQewpWM-FMo6CA",
"name" : "node1",
"transport_address" : "node1_ip:9300",
"attributes" : {
"ml.machine_memory" : "16656506880",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"weight_ranking" : 1
},
"can_remain_on_current_node" : "yes",
"can_rebalance_cluster" : "no",
"can_rebalance_cluster_decisions" : [
{
"decider" : "cluster_rebalance",
"decision" : "NO",
"explanation" : "the cluster has unassigned shards and cluster setting [cluster.routing.allocation.allow_rebalance] is set to [indices_all_active]"
}
],
"can_rebalance_to_other_node" : "no",
"rebalance_explanation" : "rebalancing is not allowed",
"node_allocation_decisions" : [
{
"node_id" : "7LIgwS6NQEuF0H8S-lLZCA",
"node_name" : "node2",
"transport_address" : "node2_ip:9300",
"node_attributes" : {
"ml.machine_memory" : "16656506880",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true"
},
"node_decision" : "no",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[private_index][0], node[7LIgwS6NQEuF0H8S-lLZCA], [R], s[STARTED], a[id=XeahuLK8QkqD0iKbKy-1sw]]"
}
]
}
]
}
```
Doing my due diligence, I see some differences:
There is a new "decider", though it is followed by an identical "cluster\_rebalance"
```
{
"decider" : "rebalance_only_when_active",
"decision" : "NO",
"explanation" : "rebalancing is not allowed until all replicas in the cluster are active"
},
```
Otherwise they look identical to me. I'm interpreting this as "Can't re-balance private\_index because not all replica shards are active." Is that correct? So it looks like I may not be gathering the right data here, because I know already that the replica shard I am trying to assign is not being properly assigned to the 2nd node. Further down it describes that the replica shard (I assume [R] refers to replica) is in INITIALIZING with reason master ... has not removed previously failed shard.
I'm going to do some googling on this `master ... has not removed previously failed shard.`
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 11, 2019, 4:47pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/6 "2019-07-11T16:47:38Z")
> [@penguinairlines](#):
>
> ```auto
> GET /_cluster/allocation/explain
> {
> "index": "private_index",
> "shard": 0,
> "primary": true
> }
>
> ```
You're asking for the status of the primary: `"primary": true`. We want to see the information about the replica, for which you need to say `"primary": false`.
Also, best to add more posts rather than to add stuff to an existing one with edits. It's just by chance that I still had this tab open, and if I hadn't then I'd never have noticed that there was more information to read.
---
### Author:  [@penguinairlines](https://discuss.elastic.co/u/penguinairlines)
#### Post date: [July 11, 2019, 5:15pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/7 "2019-07-11T17:15:13Z")
> [@DavidTurner](#):
>
> Also, best to add more posts rather than to add stuff to an existing one with edits. It's just by chance that I still had this tab open, and if I hadn't then I'd never have noticed that there was more information to read.
Oh yes, I understand. I felt I was still doing my research and did not want to pester/notify just yet.
> [@DavidTurner](#):
>
> You're asking for the status of the primary: `"primary": true` . We want to see the information about the replica, for which you need to say `"primary": false` .
Ah I see now. I ran this earlier and thought it looked the same so I skimmed right past it. I also thought ` "current_state" : "started",` couldn't possibly be true when `/_cat/shards` showed otherwise.
```
primary true:
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[private_index][0], node[7LIgwS6NQEuF0H8S-lLZCA], [R], s[STARTED], a[id=XeahuLK8QkqD0iKbKy-1sw]]"
primary false:
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[private_index][0], node[ihTj7kz6QQewpWM-FMo6CA], [P], s[STARTED], a[id=Ve28ZP0xQTqpAunmTBl7pA]]"
```
Reviewing this, the reasons appear mirrored. And as it turns out, I re-ran `GET /_cat/shards` and found that the replica is indeed now started!!! All of my custom indices are healthy now! Thank you David!
Looks like a little cleanup remains though, as `.monitoring-es-7-2019.07.0X` are not allocated and cluster health is in red, possibly because of this. I'll continue documenting my progress in the hopes that this thread can help someone else in my situation one day, but I'm hopeful I'm on the right track as things stand.
Regarding Issue 2, it appears that the indices from these 2 days have failed to allocate for the reason "stale or corrupt". Once again I will reference cluster reroute doc, specifically [Forced allocation on unrecoverable errors](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/cluster-reroute.html#_forced_allocation_on_unrecoverable_errors), which describes allocate\_stale\_primary
```
[root@node1: /etc/elasticsearch]# curl -XGET node1_ip:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep .monitoring-es
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2364 100 2364 0 0 66088 0 --:--:-- --:--:-- --:--:-- 67542
.monitoring-es-7-2019.07.09 0 p UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2019.07.09 0 r UNASSIGNED REPLICA_ADDED
.monitoring-es-7-2019.07.08 0 p UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2019.07.08 0 r UNASSIGNED REPLICA_ADDED
.monitoring-es-7-2019.07.05 0 r STARTED
.monitoring-es-7-2019.07.05 0 p STARTED
.monitoring-es-7-2019.07.11 0 p STARTED
.monitoring-es-7-2019.07.11 0 r STARTED
.monitoring-es-7-2019.07.06 0 r STARTED
.monitoring-es-7-2019.07.06 0 p STARTED
.monitoring-es-7-2019.07.10 0 p STARTED
.monitoring-es-7-2019.07.10 0 r STARTED
.monitoring-es-7-2019.07.07 0 r STARTED
.monitoring-es-7-2019.07.07 0 p STARTED
```
Attempted:
```
POST /_cluster/reroute
{
"commands" : [
{
"allocate_stale_primary" : {
"index" : ".monitoring-es-7-2019.07.08", "shard" : 0,
"node" : "node1",
"accept_data_loss" : true
}
}
]
}
```
but this results in `IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn't, current files: [write.lock]]; nested: FileNotFoundException[no segments* file found in store(ByteSizeCachingDirectory(HybridDirectory@/var/lib/elasticsearch/nodes/0/indices/dPylp9ewSdiKnwsPzD7Z4g/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2846a2ac)): files: [write.lock]]; ",`
I believe in my case, these have gotten corrupt, so I am not looking for a command to destroy these. They do not exist in the GUI, I'm assuming due to the hidden element of the file name.
> **[Delete index API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html)**
Simple enough, I was successful with that. `DELETE /.monitoring-es-7-2019.07.09`
And `GET /_cluster/health` now returns "green"!!!!! I can no longer spot any errors in my environment, so I believe the above is all that's needed for a clean recovery! Now to go lecture my fellow co-worker for uninstall logstash everywhere with chef 😉
Thanks once again to @DavidTurner you've been invaluable in this process, in which I was most definitely stuck!
---
### Author:  [@penguinairlines](https://discuss.elastic.co/u/penguinairlines)
#### Post date: [July 11, 2019, 5:44pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/8 "2019-07-11T17:44:34Z")
Thank you David! I'm all wrapped up and ready to be put under my boss's christmas tree!
btw, is there any way to flag this as resolved or no worries there?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 11, 2019, 6:12pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/9 "2019-07-11T18:12:50Z")
No probs, glad we could help. I've marked your last post as a solution - I think there's a way to do that yourself but I only get the moderators UI so I've no idea where the button would be for you.
---
### Author:  [@penguinairlines](https://discuss.elastic.co/u/penguinairlines)
#### Post date: [July 11, 2019, 7:48pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/10 "2019-07-11T19:48:30Z")
Ah, I now see the option to mark a solution. I was looking at the original post though, to mark/tag the thread as solved/resolved, but that doesn't appear to be a feature of this forum.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 8, 2019, 7:48pm UTC](https://discuss.elastic.co/t/indices-stuck-after-recovery-from-backup/189843/11 "2019-08-08T19:48:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.