# Elasticsearch Cluster health is RED
**URL:** https://discuss.elastic.co/t/elasticsearch-cluster-health-is-red/351622
**Category:** Elasticsearch
**Created:** [January 23, 2024, 12:05pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-health-is-red/351622 "2024-01-23T12:05:03Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)
#### Post date: [January 23, 2024, 12:05pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-health-is-red/351622/1 "2024-01-23T12:05:03Z")
Hi Team,
We had Elasticsearch with two node and due to some infra issues the server went down . Once it become online , I started the service and it was successful. But while checking the log showing below error. Could you please let me know how to attach the log file to this thread
Please help to bring the cluster to healthy state.
```auto
> [2024-01-23T16:08:03,947][INFO][o.e.c.s.ClusterApplierService] [cb-2] master node changed {previous [{cb-2}{3E6yK8J6Sy-TL9k5yDGi-w}{bKpu6L7BRV-xMMWPltISCg}{cb-2}{10.10.18.174}{10.10.18.174:9300}{cdfhilmrstw}{8.9.2}], current []}, term: 309673, version: 313060, reason: becoming candidate: Publication.onCompletion(false)
> [2024-01-23T16:08:03,947][INFO][o.e.c.f.AbstractFileWatchingService] [cb-2] shutting down watcher thread
> [2024-01-23T16:08:03,948][INFO][o.e.c.f.AbstractFileWatchingService] [cb-2] watcher service stopped
> [2024-01-23T16:08:03,947][WARN][o.e.c.s.MasterService] [cb-2] failing [node-join[{cb-3}{jLNpa0ChRZaMTtYMdO7Jgw}{JXpDZ9VQQT-5Ue9ZmjSCig}{cb-3}{10.10.18.215}{10.10.18.215:9300}{cdfhilmrstw}{8.9.2} rejoining]]: failed to commit cluster state version [313061]
> org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication of cluster state version [313061] in term [309673] failed [committed={}]
> at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:2009) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:285) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:197) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:96) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1915) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:165) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:390) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.cluster.coordination.Coordinator$8.onFailure(Coordinator.java:1646) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:92) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:966) ~[elasticsearch-8.9.2.jar:?]
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) ~[elasticsearch-8.9.2.jar:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
> at java.lang.Thread.run(Thread.java:1623) ~[?:?]
> Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: Input/output error
```
Thanks,
Debasis
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 23, 2024, 12:25pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-health-is-red/351622/2 "2024-01-23T12:25:03Z")
> [@Debasis\_Mallick](#):
>
> `java.io.IOException: Input/output error`
The error message `Input/output error` comes from the OS and normally means there's some problem with that node's storage. I'd suggest looking at its kernel logs with `dmesg` to confirm, and replace any faulty/suspect hardware before doing much more.
---
### Author:  [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)
#### Post date: [January 24, 2024, 5:16am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-health-is-red/351622/3 "2024-01-24T05:16:16Z")
Thanks @DavidTurner for prompt response . Still I can see some I/O error even though I can write that particular mount point manually. I am checking with IT team for this error.
Thanks,
Debasis
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 24, 2024, 8:02am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-health-is-red/351622/4 "2024-01-24T08:02:17Z")
Yeah `Input/output error` doesn't imply that the whole disk/partition/mount point is unwriteable, it could be a very localized problem.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 21, 2024, 8:03am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-health-is-red/351622/5 "2024-02-21T08:03:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.