What if my storage is full?

gauravbhutani30 · August 3, 2017, 7:42am

Hi Everyone,

We are running a elastisearch cluster of 3 nodes and we have mounted our storage on SAN.
I was just wandering if the storage is full, what will happen to our cluster?

We wanted to test this scenario. Please help what are the possibilities in this kind case, what will be the solution and how we can replicate and test this scenario?

Thanks in advance!!
Gaurav

dadoonet · August 3, 2017, 8:30am

From 6.0.0-beta1 (not available yet), this will be put your indices as readonly.

github.com/elastic/elasticsearch

Switch indices read-only if a node runs out of disk space

elastic:master ← s1monw:issues/24299

opened 04:50PM - 04 Jul 17 UTC

s1monw

+301 -190

Today when we run out of disk all kinds of crazy things can happen and nodes ar…e becoming hard to maintain once out of disk is hit. While we try to move shards away if we hit watermarks this might not be possible in many situations. Based on the discussion in #24299 this change monitors disk utiliation and adds a floodstage watermark that causes all indices that are allocated on a node hitting the floodstage mark to be switched read-only (with the option to be deleted). This allows users to react on the low disk situation while subsequent write requests will be rejected. Users can switch individual indices read-write once the situation is sorted out. There is no automatic read-write switch once the node has enough space. This requires user interaction. The floodstage watermark is set to `95%` utilization by default. Closes #24299

Before this version, you will have to suffer from some troubles.
Best thing to do is to have a monitoring and stop injecting data when your nodes are running low on space.

Not related to your question but:

we have mounted our storage on SAN

This is not what we recommend. We actively recommend using local disks instead.

gauravbhutani30 · August 3, 2017, 1:47pm

Hi David,

Thanks for the response!
Does elasticsearch provides any API/mechanism from which we can monitor the disk usage?

Thanks
Gaurav

dadoonet · August 3, 2017, 3:10pm

x-pack monitoring does that.
Note that it requires a licence. The Basic (free) one is enough.

gauravbhutani30 · August 4, 2017, 9:10am

Ok...i see that x-pack has one month free subscription only.

dadoonet · August 4, 2017, 9:36am

No. The Basic license is for one year and is free.

This one month trial is to test all x-pack features. But as soon as you add your basic license the trial stops.

https://register.elastic.co/

gauravbhutani30 · August 4, 2017, 10:23am

Ok. Let me try.
Thanks for all your help

system · September 1, 2017, 10:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.