Spark is unable to write to ES when ElasticSearch is in Red state

Elastic Stack Elasticsearch
1

Hi,

Spark is unable to write to ES when the cluster is in Red state even when the index to which the spark is reading/writing is 100% available, in Green state.

The Spark job just hangs, it doesn't fail or timeout

Is there any setting? Do I need to change? The setting I see is the write will be impacted only when the index to which we are reading/writing is in a red state!

https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html

Any help would be appreciated!
Regards,
Amit

2

If the entire cluster is in a red state you probably do not have an elected master node. In this case you will not be able to write to the cluster as it could lead to data loss. This is therefore expected.

Why is your cluster in a red state in the first place?

3

One of the index primary shard was unassigned .. and that's why the cluster was in RED state