100% disk, single node cluster how to fix?

Elastic Stack Elasticsearch
1

I have a test single node cluster.
I know what the problem is but can't seems to figure out how to get out of it and fix without removing everything and star over

this node has all index without replica
because I executed this while back

PUT /_all/_settings
{
    "index" : {
        "number_of_replicas" : 0
    }
}

now because / disk is full which is also elastic data path and I can't increase disk size. how do I remove some unwanted or large older index?

because I can't connect to it even via command prompt

curl  -XGET  node11:9200/_cluster/health?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

and I know my elasticsearch/node folder has all the data.

Is there a solution in this situation?

2

What is the results of

_cat/indices/?s=pri.store.size:desc

Can you free up other non elastic disk space?

Even a few GBs might help?

3

HI
I could and I did to bring system backup

but now I want to know how to recover from this situation if there is no other way to remove any other log/files from system. and If want to remove some index to free up space?

4

Did you run the cat indices before you cleaned up?

Did you get a response?

The. Only way to clean up AFAIK is to DELETE indices directly.

5

I did further test by filling up disk to 100% manually and I can't run any elastic command at all, can't communicate with cluster at all.
everything gives me same error message.

what I believe is if you ever get to 100% and can't remove anything then only option is to go in Elasticsearch data dir and remove some indices directory or lucent files by looking up date creation of that. which is very old fashion but works ( I tested it)
then once system is up you will see that indice is unavailable. and delete it from kibana.

in production I don't want to be on that spot.

6

That is the most important statement in this whole thread :slight_smile:

7

yep. I had some check already in place but didn't know that, this can cause lot of trouble.

now what I have also thought is to put 1 or 2 gig dummy file on data dir. as storage is cheap and if it ever gets fill up we can remove that dummy file and get going to do require fix.

1 Like
8

You can / should set up stack management alerting to let you know when you get to 80% or 90%...

1 Like
9

It really depends on how you are going to do in production.

If you are self-manage everything with VMs I would say that a good approach is that every data node has a different disk for data, and in this case storage is not that cheap.

Depending on the size of the disk it can cost you the same price as the VM if you are running on cloud.

Also, with a separate data disk it makes easier to configure and control the watermarks, correctly configuring the watermarks will help you to not have this issue in production.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.