Elasticsearch node is at 100% disk usage, unable to edit configuration

[Sanskar_Panchal]

Hi,

One of my Elasticsearch instance is at 100% disk usage.

Which has resulted into NODE_LEFT.
Which then caused " This cluster has 39 unavailable primaries, 101 unavailable replicas."

And cluster is now at red health status.

I tried increasing disk size per zone , but any configuration change is not applied. (maybe because of red cluster health)

I tried deleting some non useful indices in hot_tier and also reduced no. of replicas for some indices for nothing seems reduce space in the full node.

The full disk node is also not visible when _cat/nodes is run:

GET _cat/nodes?h=id,name,r,u,du

OUTPUT:
nggi instance-0000000022   himrst 110.3d  68.1gb
Xy2c instance-0000000023   rw      93.6d 237.8gb
BoE0 instance-0000000024   cr      82.5d  27.5mb
jfxF tiebreaker-0000000017 mv     156.2d    24mb
BvYm instance-0000000016   rw     177.7d 237.8gb

How can I resolve this ?

[stephenb (Stephen Brown)]

@Sanskar_Panchal

You should open a support ticket through Elastic Support.

[Sanskar_Panchal]

Actually i don't have the required subscribtion level to open a case in elastic support.
That button is greyed out for me.

[stephenb (Stephen Brown)]

Exactly what product are you using?

Elastic Cloud?

cloud.elastic.co?

[Sanskar_Panchal]

Yes , I am using https://cloud.elastic.co and I am at Standard subscription level.

[stephenb (Stephen Brown)]

Official Elasticsearch Pricing: Elastic Cloud, Managed Elasticsearch

Someone in your org has support Support comes with Standard see page and below.

Standard
SUPPORT
Web-based support
2 Support Contacts
3-business day target response time (Elastic Cloud only)

Anyway

Try to set the bad node to Stop Routing it in the 3 little dots.

Then try to scale nodes to 8 GB

Or if stopped routing already try to start routing and delete indices.

See if that will work.

But finding who has your support contact and filing ticket is probably Best