# Cluster red, unassigned shards, no response on writes
**URL:** https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114
**Category:** Elasticsearch
**Created:** [December 28, 2022, 4:38pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114 "2022-12-28T16:38:00Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@decibel83](https://discuss.elastic.co/u/decibel83)
#### Post date: [December 28, 2022, 4:38pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/1 "2022-12-28T16:38:00Z")
Hi everyone,
I'm dealing with a problem on Elasticsearch 7.17.5 (I've also tried to upgrade it to version 8 but I'm having the same problem with that version too) which is hanged, red status and unresponsive on any write request (for example POST and DELETE calls).
I tried to enlarge the Java heap size, restarting the server many times, with no success.
This is the `_cluster/health` output:
```auto
{
"cluster_name": "my_elastic_cluster",
"status": "red",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 234,
"active_shards": 234,
"relocating_shards": 0,
"initializing_shards": 1,
"unassigned_shards": 125,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 252,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 26331,
"active_shards_percent_as_number": 65
}
```
Here you can find some information and logs:
- `/_cat/indices?v` output: [health status index uuid pri rep - Pastebin.com](https://pastebin.com/g77jhwwL)
- `/_cat/shards?v` output: [index shard prirep state - Pastebin.com](https://pastebin.com/nUTq9jf3)
- `/var/log/elasticsearch/my_elastic_cluster.log` file: [[2022-12-28T16:14:53,129][INFO][o.e.n.Node] [elastic.my.domain.c - Pastebin.com](https://pastebin.com/YmcGTrsn)
Could you help me to find out what's going on, and how to resume this server without loosing data, please?
Thank you very much!
Bye
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 28, 2022, 5:50pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/2 "2022-12-28T17:50:23Z")
How many nodes do you have?
Which version are you using? The log you shared says it is `7.17.7`.
> [2022-12-28T16:14:53,129][INFO][o.e.n.Node] [[elastic.my.domain.com](http://elastic.my.domain.com)] version[7.17.7]
Also, this is everything you have in the log? I didn't find anything that could give a hint of your issue.
What is the status of the following request:
```auto
GET _cluster/allocation/explain
```
---
### Author:  [@decibel83](https://discuss.elastic.co/u/decibel83)
#### Post date: [December 28, 2022, 6:46pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/3 "2022-12-28T18:46:51Z")
> [@leandrojmp](#):
>
> How many nodes do you have?
This is a single node cluster dedicated to beats, with the following configuration:
```auto
bootstrap.memory_lock: false
cluster.initial_master_nodes:
- elastic.my.domain.com
cluster.name: my_elastic_cluster
discovery.seed_hosts:
- elastic.my.domain.com
http.port: '9200'
network.bind_host:
- 123.123.123.123
- 127.0.0.1
- localhost
network.host: 123.123.123.123
node.data: true
node.master: true
node.name: elastic.my.domain.com
transport.port: '9301'
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
action.auto_create_index: true
```
> [@leandrojmp](#):
>
> Which version are you using? The log you shared says it is `7.17.7`.
Yes, the version is **7.17.7** , anyway I've already tried to upgrade to version 8 on this machine but I had the same problem, so I rolled back to 7.17.7.
> [@leandrojmp](#):
>
> Also, this is everything you have in the log? I didn't find anything that could give a hint of your issue.
That's all, I don't have any more lines in the log.
> [@leandrojmp](#):
>
> What is the status of the following request:
>
> ```auto
> GET _cluster/allocation/explain
>
> ```
This is the output:
```auto
{
"note": "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
"index": "auditbeat-7.17.5-2022.11.28-000027",
"shard": 0,
"primary": true,
"current_state": "unassigned",
"unassigned_info": {
"reason": "CLUSTER_RECOVERED",
"at": "2022-12-28T16:15:07.145Z",
"last_allocation_status": "throttled"
},
"can_allocate": "yes",
"allocate_explanation": "can allocate the shard",
"target_node": {
"id": "94z2vzjeTPGgIWheedCseA",
"name": "elastic.my.domain.com",
"transport_address": "123.123.123.123:9301",
"attributes": {
"ml.machine_memory": "16821936128",
"xpack.installed": "true",
"transform.node": "true",
"ml.max_open_jobs": "512",
"ml.max_jvm_size": "8589934592"
}
},
"allocation_id": "V_KwpEg6RkihWwMGTHKiRA",
"node_allocation_decisions": [
{
"node_id": "94z2vzjeTPGgIWheedCseA",
"node_name": "elastic.my.domain.com",
"transport_address": "123.123.123.123:9301",
"node_attributes": {
"ml.machine_memory": "16821936128",
"xpack.installed": "true",
"transform.node": "true",
"ml.max_open_jobs": "512",
"ml.max_jvm_size": "8589934592"
},
"node_decision": "yes",
"store": {
"in_sync": true,
"allocation_id": "V_KwpEg6RkihWwMGTHKiRA"
}
}
]
}
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 28, 2022, 6:57pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/4 "2022-12-28T18:57:05Z")
What is the result of:
```auto
GET _cat/health?v
```
Also, choose any `UNASSIGNED` index and run the following:
```auto
GET /_cluster/allocation/explain
{
"index": "index-name",
"shard": 0,
"primary": true
}
```
It looks that your cluster is still bringing up your indices and you just need to wait.
---
### Author:  [@decibel83](https://discuss.elastic.co/u/decibel83)
#### Post date: [December 28, 2022, 7:18pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/5 "2022-12-28T19:18:08Z")
> [@](#):
>
> What is the result of:
>
> ```auto
> GET _cat/health?v
>
> ```
```auto
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1672254759 19:12:39 my_elastic_cluster red 1 1 234 234 0 1 125 305 2.9h 65.0%
```
Also, choose any `UNASSIGNED` index and run the following:
```auto
GET /_cluster/allocation/explain
{
"index": "index-name",
"shard": 0,
"primary": true
}
```
```auto
TEST root@elk ~# curl -X GET "localhost:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d'
{
"index": "auditbeat-7.17.5-2022.11.28-000027",
"shard": 0,
"primary": true
}
'
{
"index" : "auditbeat-7.17.5-2022.11.28-000027",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2022-12-28T16:15:07.145Z",
"last_allocation_status" : "throttled"
},
"can_allocate" : "yes",
"allocate_explanation" : "can allocate the shard",
"target_node" : {
"id" : "94z2vzjeTPGgIWheedCseA",
"name" : "elastic.my.domain.com",
"transport_address" : "123.123.123.123:9301",
"attributes" : {
"ml.machine_memory" : "16821936128",
"xpack.installed" : "true",
"transform.node" : "true",
"ml.max_open_jobs" : "512",
"ml.max_jvm_size" : "8589934592"
}
},
"allocation_id" : "V_KwpEg6RkihWwMGTHKiRA",
"node_allocation_decisions" : [
{
"node_id" : "94z2vzjeTPGgIWheedCseA",
"node_name" : "elastic.my.domain.com",
"transport_address" : "123.123.123.123:9301",
"node_attributes" : {
"ml.machine_memory" : "16821936128",
"xpack.installed" : "true",
"transform.node" : "true",
"ml.max_open_jobs" : "512",
"ml.max_jvm_size" : "8589934592"
},
"node_decision" : "yes",
"store" : {
"in_sync" : true,
"allocation_id" : "V_KwpEg6RkihWwMGTHKiRA"
}
}
]
}
```
> [@](#):
>
> It looks that your cluster is still bringing up your indices and you just need to wait.
Please note that this server is into this situation from some days...
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 28, 2022, 8:30pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/6 "2022-12-28T20:30:29Z")
What does `GET _nodes/hot_threads?threads=9999` say?
---
### Author:  [@decibel83](https://discuss.elastic.co/u/decibel83)
#### Post date: [January 5, 2023, 3:14pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/7 "2023-01-05T15:14:56Z")
> [@DavidTurner](#):
>
> What does `GET _nodes/hot_threads?threads=9999` say?
Hi David, sorry for my late answer!
I tried that call but it remains hanged without any answer even after many minutes:
```auto
TEST me@elk ~> curl -X GET "localhost:9200/_nodes/hot_threads?threads=9999"
...
```
I see no new error logs when I try to make this call.
This is the very same behaviour when I tried to delete some data or a index from this Elasticsearch instance...
---
### Author:  [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)
#### Post date: [January 5, 2023, 3:32pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/8 "2023-01-05T15:32:48Z")
@decibel83 what does `GET _cat/allocation?v` returns ?
---
### Author:  [@decibel83](https://discuss.elastic.co/u/decibel83)
#### Post date: [January 5, 2023, 3:47pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/9 "2023-01-05T15:47:05Z")
> [@Ayush\_Mathur](#):
>
> @decibel83 what does `GET _cat/allocation?v` returns ?
No answer here too:
```auto
TEST me@elk ~> curl -X GET "localhost:9200/_cat/allocation?v"
```
---
### Author:  [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)
#### Post date: [January 5, 2023, 4:01pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/10 "2023-01-05T16:01:44Z")
@decibel83 can you please run: `curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED`
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 5, 2023, 5:07pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/11 "2023-01-05T17:07:15Z")
> [@decibel83](#):
>
> I tried that call but it remains hanged without any answer even after many minutes:
That is interesting. Could you capture a thread dump of the elected master with `jstack`?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 5, 2023, 5:21pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/12 "2023-01-05T17:21:02Z")
> [@DavidTurner](#):
>
> That is interesting. Could you capture a thread dump of the elected master with `jstack`?
On reflection I'd probably want to see thread dumps from all nodes. I note that you have only one node, so of course that's the elected master and the one you should dump, but I'm just saying this for the sake of future readers who might encounter something similar.
---
### Author:  [@decibel83](https://discuss.elastic.co/u/decibel83)
#### Post date: [January 25, 2023, 1:28pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/13 "2023-01-25T13:28:10Z")
> [@Ayush\_Mathur](#):
>
> curl -XGET localhost:9200/\_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
```auto
me@elk:/home/me# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 30942 100 30942 0 0 570k 0 --:--:-- --:--:-- --:--:-- 570k
auditbeat-7.17.5-2022.11.28-000027 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.04.08-000021 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.01.18-000013 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.07.20 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-error-000009 0 r UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.03.09-000018 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.01.19-000013 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.02.08-000015 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-ilm-history-5-2022.11.25-000025 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.02.17-000016 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.11.28-000056 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.13.1-000014 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.02.07-000015 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.17.0-2022.03.29-000005 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.10.26-000009 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.03.10-000018 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.02.07-000015 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.12.1-000017 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.01.29-000014 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.11.21-000058 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-ilm-history-5-2022.08.27-000019 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.11.28-000059 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.11.09-000010 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.03.10-000018 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-ilm-history-5-2022.10.26-000023 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.10.19 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.17.0-2022.11.28-000041 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.08.03-000003 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.11.18-000057 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.11.29 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.11.29 0 r UNASSIGNED CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.03.16 0 p UNASSIGNED CLUSTER_RECOVERED
metricbeat-7.14.1-2022.11.28-001899 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.11.28-000056 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.03.29-000020 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.03.30-000020 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.03.30-000020 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.02.27-000017 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.17.0-2022.04.08-000006 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.17.0-2022.03.19-000004 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.01.08-000012 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.17.5-2022.11.28-000030 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.01.18-000013 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.14.1-000014 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.03.20-000019 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.17.5-000004 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.17.6-2022.11.29-000012 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.17.0-2022.02.16-000001 0 p UNASSIGNED CLUSTER_RECOVERED
metricbeat-7.17.5-2022.11.29-000710 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.12.15 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.02.27-000017 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.02.18-000016 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.01.08-000012 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.03.20-000019 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.03.19-000019 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.11.16-000057 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-ilm-history-5-2022.09.26-000021 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.03.09 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.12.1-000019 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.01.09-000012 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.02.27-000017 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.17.5-onboarding-2022.07.12 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.17.5-2022.11.23-000029 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.11.26-000059 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.02.28-000017 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.03.29-000020 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.01.28-000014 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.02.07-000015 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.07.20-000002 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.17.5-000003 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.03.19-000019 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.03.19-000019 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-profile-000009 0 r UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.17.5-000002 0 p UNASSIGNED CLUSTER_RECOVERED
metricbeat-7.6.1-2022.11.29-000209 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.01.09-000012 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.14.1-000015 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.07.06-000001 0 p UNASSIGNED CLUSTER_RECOVERED
metricbeat-7.17.0-2022.11.29-000140 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.02.18-000016 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.01.28-000014 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.01.19-000013 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.12.1-000020 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.04.08-000021 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.03.29-000020 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.01.29-000014 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.01.08-000012 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.17.5-2022.11.28-000030 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.12.02 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.02.28-000017 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.13.1-000016 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.09.28-000007 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.11.28-000059 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.17.0-2022.02.26-000002 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.13.1-000015 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.01.28-000014 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.14.1-000013 0 p UNASSIGNED CLUSTER_RECOVERED
metricbeat-7.17.6-2022.11.28-000028 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.10.12-000008 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.07.12 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.08.17-000004 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.6.1-2022.11.23-000058 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.17.5-2022.11.18-000028 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.17.5-000005 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-metric-000009 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.08.31-000005 0 p UNASSIGNED CLUSTER_RECOVERED
heartbeat-7.14.1-2022.03.09-000018 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.12.01 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.17.0-2022.03.09-000003 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.13.1-000013 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.12.1-000018 0 p UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.02.17-000016 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-transaction-000009 0 r UNASSIGNED CLUSTER_RECOVERED
filebeat-7.14.1-2022.03.09-000018 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.6.1-2022.02.08-000015 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.02.17-000016 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-span-000009 0 r UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.03.22 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.10.18 0 p UNASSIGNED CLUSTER_RECOVERED
auditbeat-7.14.1-2022.01.18-000013 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.11.23-000011 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.11.23-000011 0 p UNASSIGNED CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.11.29 0 p UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.09.14-000006 0 p UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.14.1-000012 0 p UNASSIGNED CLUSTER_RECOVERED
```
---
### Author:  [@decibel83](https://discuss.elastic.co/u/decibel83)
#### Post date: [January 25, 2023, 1:36pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/14 "2023-01-25T13:36:39Z")
> [@DavidTurner](#):
>
> That is interesting. Could you capture a thread dump of the elected master with `jstack`?
```auto
root@elk:~# ps ax|grep elastic
1774 ? Ssl 73:33 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -Xms8g -Xmx8g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=4294967296 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
1944 ? Sl 0:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
```
Here you can find the output of the command `/usr/share/elasticsearch/jdk/bin/jstack 1944` : [https://pastebin.com/H8cqLLtz](https://pastebin.com/H8cqLLtz).
Thank you!
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 25, 2023, 1:59pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/15 "2023-01-25T13:59:29Z")
> [@decibel83](#):
>
> [https://pastebin.com/H8cqLLtz](https://pastebin.com/H8cqLLtz).
I get this error:
> Error, this is a private paste or is pending moderation. If this paste belongs to you, please login to Pastebin to view it.
Try [https://gist.github.com](https://gist.github.com) instead?
---
### Author:  [@decibel83](https://discuss.elastic.co/u/decibel83)
#### Post date: [January 25, 2023, 2:33pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/16 "2023-01-25T14:33:07Z")
Here it is: [jjstack output · GitHub](https://gist.github.com/mmartinello/421bef95b3bbae5b0321127706a736d4)
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 25, 2023, 2:41pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/17 "2023-01-25T14:41:58Z")
Thanks. This indicates you're hitting this bug:
>
>
> \*\*Elasticsearch version\*\* (\`bin/elasticsearch --version\`): 7.10.1
>
> \*\*Plugins i…nstalled\*\*: Cloud
>
> \*\*JVM version\*\* (\`java -version\`): 15.0.1+9
>
> \*\*OS version\*\* (\`uname -a\` if on a Unix-like system): Cloud
>
> \*\*Description of the problem including expected versus actual behavior\*\*:
>
> All generic threads are waiting with the following stack trace:
>
> \`\`\`
> at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:211)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;IZZZJ)I (AbstractQueuedSynchronizer.java:714)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(I)V (AbstractQueuedSynchronizer.java:937)
> at java.util.concurrent.locks.ReentrantLock$Sync.lock()V (ReentrantLock.java:153)
> at java.util.concurrent.locks.ReentrantLock.lock()V (ReentrantLock.java:322)
> at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(Lorg/elasticsearch/action/DocWriteRequest;)V (BulkProcessor.java:379)
> at org.elasticsearch.action.bulk.BulkProcessor.add(Lorg/elasticsearch/action/DocWriteRequest;)Lorg/elasticsearch/action/bulk/BulkProcessor; (BulkProcessor.java:361)
> at org.elasticsearch.action.bulk.BulkProcessor.add(Lorg/elasticsearch/action/index/IndexRequest;)Lorg/elasticsearch/action/bulk/BulkProcessor; (BulkProcessor.java:347)
> at org.elasticsearch.xpack.ilm.history.ILMHistoryStore.lambda$putAsync$0(Lorg/elasticsearch/action/index/IndexRequest;Lorg/elasticsearch/xpack/ilm/history/ILMHistoryItem;)V (ILMHistoryStore.java:150)
> at org.elasticsearch.xpack.ilm.history.ILMHistoryStore$$Lambda$8091+0x0000000801eda370.run()V (Unknown Source)
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run()V (ThreadContext.java:678)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1130)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:630)
> at java.lang.Thread.run()V (Thread.java:832)
> \`\`\`
>
> Meanwhile one of the scheduler threads is holding the lock on which they're waiting, and is blocked here:
>
> \`\`\`
> jdk.internal.misc.Unsafe.park(ZJ)V (Native Method)
> java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:211)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;IZZZJ)I (AbstractQueuedSynchronizer.java:714)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(I)V (AbstractQueuedSynchronizer.java:1046)
> java.util.concurrent.Semaphore.acquire()V (Semaphore.java:318)
> org.elasticsearch.action.bulk.BulkRequestHandler.execute(Lorg/elasticsearch/action/bulk/BulkRequest;J)V (BulkRequestHandler.java:59)
> org.elasticsearch.action.bulk.BulkProcessor.execute(Lorg/elasticsearch/action/bulk/BulkRequest;J)V (BulkProcessor.java:454)
> org.elasticsearch.action.bulk.BulkProcessor.execute()V (BulkProcessor.java:463)
> org.elasticsearch.action.bulk.BulkProcessor.access$400(Lorg/elasticsearch/action/bulk/BulkProcessor;)V (BulkProcessor.java:54)
> org.elasticsearch.action.bulk.BulkProcessor$Flush.run()V (BulkProcessor.java:503)
> org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun()V (Scheduler.java:213)
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run()V (AbstractRunnable.java:37)
> java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (Executors.java:515)
> java.util.concurrent.FutureTask.run()V (FutureTask.java:264)
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V (ScheduledThreadPoolExecutor.java:304)
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1130)
> java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:630)
> java.lang.Thread.run()V (Thread.java:832)
> \`\`\`
>
> The semaphore on which it is waiting is held, apparently, by another ongoing flush. I haven't chased this any further but I could believe that the ongoing flush needs a generic thread to make progress.
>
> \*\*Steps to reproduce\*\*:
>
> Unknown.
>
> \*\*Provide logs (if relevant)\*\*:
>
> Not available, but I can share a heap dump privately.
>
> \*\*Workaround\*\*:
>
> The deadlocked node must be restarted, it will not recover on its own. If the issue persists then the problematic component can be disabled by setting \`indices.lifecycle.history\_index\_enabled: false\` in the \`elasticsearch.yml\` file on each master-eligible node and then restarting them all.
A fix is planned for 8.7 but until then I suggest you use the workaround described in the issue.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 22, 2023, 2:42pm UTC](https://discuss.elastic.co/t/cluster-red-unassigned-shards-no-response-on-writes/322114/18 "2023-02-22T14:42:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.