# Red Cluster Health - Unsure How to Fix
**URL:** https://discuss.elastic.co/t/red-cluster-health-unsure-how-to-fix/326464
**Category:** Elasticsearch
**Created:** [February 24, 2023, 6:32pm UTC](https://discuss.elastic.co/t/red-cluster-health-unsure-how-to-fix/326464 "2023-02-24T18:32:36Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@bcantrell](https://discuss.elastic.co/u/bcantrell)
#### Post date: [February 24, 2023, 6:32pm UTC](https://discuss.elastic.co/t/red-cluster-health-unsure-how-to-fix/326464/1 "2023-02-24T18:32:36Z")
Hi all,
I have been trying to figure out a problem where my Kibana is not able to keep connections alive with the Elasticsearch instance, and I think it is because of red cluster/index health. When Kibana is running, I get the below messages spammed constantly in the Elasticsearch cluster logs on all nodes. This started the same day that one of the Elasticsearch agents crashed and has been continuing since.
> [2023-02-24T08:34:56,661][WARN][o.e.c.r.a.AllocationService] [NodeName] [.kibana-event-log-8.1.3-000010][0] marking unavailable shards as stale: [V\_z\_SX6bTbyL3xN\_dUJXdA]
> [2023-02-24T08:35:02,032][WARN][o.e.c.r.a.AllocationService] [NodeName] [.ds-auditbeat-8.1.3-2023.01.28-000014][0] marking unavailable shards as stale: [y5ut-1ZZTVejQgoSiL5XZQ]
> [2023-02-24T08:35:02,212][WARN][r.suppressed] [NodeName] path: /winlogbeat-_,logs-endpoint.events._,logs-windows._/\_eql/search, params: {allow\_no\_indices=true, index=winlogbeat-_,logs-endpoint.events._,logs-windows._}
> org.elasticsearch.action.search.SearchPhaseExecutionException: start
> at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.onPhaseFailure(CanMatchPreFilterSearchPhase.java:465) [elasticsearch-8.1.3.jar:8.1.3]
> at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.onFailure(CanMatchPreFilterSearchPhase.java:454) [elasticsearch-8.1.3.jar:8.1.3]
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-8.1.3.jar:8.1.3]
> at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-8.1.3.jar:8.1.3]
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) [elasticsearch-8.1.3.jar:8.1.3]
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.1.3.jar:8.1.3]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
> at java.lang.Thread.run(Thread.java:833) [?:?]
> Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.ds-winlogbeat-8.1.3-2023.01.28-000013][2]]. Consider using `allow_partial_search_results` setting to bypass this error.
> at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.checkNoMissingShards(CanMatchPreFilterSearchPhase.java:216) ~[elasticsearch-8.1.3.jar:8.1.3]
> at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase.run(CanMatchPreFilterSearchPhase.java:140) ~[elasticsearch-8.1.3.jar:8.1.3]
> at org.elasticsearch.action.search.CanMatchPreFilterSearchPhase$1.doRun(CanMatchPreFilterSearchPhase.java:459) ~[elasticsearch-8.1.3.jar:8.1.3]
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.1.3.jar:8.1.3]
> ... 6 more
This is a massive block that I can provide the whole error for, if necessary:
> [2023-02-24T08:32:30,904][ERROR][o.e.x.e.p.RestEqlSearchAction] [NodeName] failed to send failure response
> java.lang.IllegalStateException: Channel is already closed
> ...
> Suppressed: java.lang.IllegalArgumentException: reader id must be specified
I randomly was able to log in to Kibana once after starting it up, but before it lost connection again, and saw that the .ds-winlogbeat-8.1.3-2023.01.28-000013 index is at Red health. I have found plenty of information on how to fix red health issues, but here is my problem: I cannot keep connected in Kibana for more than a few seconds (if that), and I do not have an API key generated aside from the limited permissions for beat agents. My feeling is that the red health is the root cause, but I am stumped at this point and desparate for any suggestions.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 25, 2023, 9:35am UTC](https://discuss.elastic.co/t/red-cluster-health-unsure-how-to-fix/326464/2 "2023-02-25T09:35:02Z")
> [@bcantrell](#):
>
> here is my problem: I cannot keep connected in Kibana for more than a few seconds (if that), and I do not have an API key generated aside from the limited permissions for beat agents.
You won't be able to work out the problem without getting responses from Elasticsearch APIs. If Kibana is having trouble talking to Elasticsearch then I suggest bypassing it and talking to Elasticsearch directly, using `curl` or similar. If security is enabled then you'll need to include credentials:
```auto
curl -u 'USERNAME:PASSWORD' https://host:port/...
```
If the cluster is in such bad health that it cannot even check your credentials (e.g. the `.security` index is missing) then you will need to set up a user in the [file realm](https://www.elastic.co/guide/en/elasticsearch/reference/current/file-realm.html) instead.
---
### Author:  [@bcantrell](https://discuss.elastic.co/u/bcantrell)
#### Post date: [February 27, 2023, 1:49pm UTC](https://discuss.elastic.co/t/red-cluster-health-unsure-how-to-fix/326464/3 "2023-02-27T13:49:17Z")
Thank you, David! I can CURL with my user just fine. I swear I tried that but probably fat-fingered something and moved on in my frustration. I will give it a shot from here.
---
### Author:  [@bcantrell](https://discuss.elastic.co/u/bcantrell)
#### Post date: [February 27, 2023, 10:08pm UTC](https://discuss.elastic.co/t/red-cluster-health-unsure-how-to-fix/326464/4 "2023-02-27T22:08:28Z")
For future Googlers, this came down to a corrupted transaction log.
Using this link, and this query, I found the below message in the "details" block:
> **[Red or yellow cluster status | Elasticsearch Guide \[8.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/red-yellow-cluster-status.html)**
`GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.*`
> translog from source [name/path] is corrupted
Then using this link, I was able to target and repair the corrupted index:
> **[elasticsearch-shard | Elasticsearch Guide \[master\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/shard-tool.html)**
Cluster health is back to green, Kibana is working.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 27, 2023, 10:09pm UTC](https://discuss.elastic.co/t/red-cluster-health-unsure-how-to-fix/326464/5 "2023-03-27T22:09:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.