ELK service stops when space is limited

[juancamiloll (Juan C)]

Hello,

I am working with an old version of ELK 7.6.0.

I have detected that sometimes one of the 3 services stops when the disk reaches 94% or 95% of its capacity. I don't know if this is a default setting, I don't know if some of you have had this happen.

I have 2 questions

1. Can I increase that percentage that causes the services to stop? or is it something specific to linux?

2. When I search from the root looking for the heaviest file I find that the folder is /opt/lib/elasticsearch/nodes/0/indices where some strange names appear I assume they are the same that I can see from Kibana, my question is if there is any way to debug these index? or any other file or files that I can debug in order to free up space and that is unnecessary information?

[leandrojmp (Leandro Pereira)]

@juancamiloll

I don't know if this is a default setting

This is a default setting, Elasticsearch has 3 disk watermarks, low, high and flood, and their default values are 85%, 90% and 95%, you can read more about it here in the documentation.

Basically when a node reach the flood stage watermark, 95% of disk used, Elasticsearch will mark every indice that has at least one shard on the node as read only, so no data can be written to those indices until the flood stage is solved.

You can change their settings, increase or decrease the watermark if you need/want.

@juancamiloll

When I search from the root looking for the heaviest file I find that the folder is /opt/lib/elasticsearch/nodes/0/indices where some strange names appear I assume they are the same that I can see from Kibana, my question is if there is any way to debug these index?

You should not not make any change to those files direct on the disk, if you do you may corrupt your indices and lost data.

If you need to delete indices to clean space you need to do that using Kibana or the Elasticsearch API, never touch the files on the disk.

[juancamiloll (Juan C)]

@leandrojmp Thank you very much for your answer, it really helped me a lot. The disk is 8TB and I am short of space, would you recommend me to leave the default configuration? do you think I can cause any problem if I raise that 95% to 97% or 98%?

In case you don't see any problem, where do you apply that change? from Kibana or modifying some parameter in some specific configuration file (which is the path)?

[leandrojmp (Leandro Pereira)]

@juancamiloll

do you think I can cause any problem if I raise that 95% to 97% or 98%?

It will give you some room, but will not solve your problem, you need to remove some data or increase your disk.

Check your indices and see if you can remove anything.

@juancamiloll

where do you apply that change?

You can make a request in Kibana Dev tools, the documentation I shared has an example in the end.