Ancient Broken ElasticSearch question

Elastic Stack Elasticsearch
1

I have inherited an ancient (1.4, I think) version of single node elasticstack running (well, not running at the moment).
My original plan was to logon and clear out indexes because the customer got a low disk warning. When I logged on Elastic wasn't running. I tried starting it so I could remove old indexes to clean up space. Now I cannot get elastic to actually start.
I am unable to get any information on why elastic search won't stay running. When I start elasticsearch java kicks off and the process starts. After about 30 seconds it dies and leaves a pid file behind (not sure if that is relevant) and the log file (/var/log/elasticsearch/elasticstack.log) is literally empty. I checked permissions all around nothing has changed and permissions appear as one would expect. The partition that houses the elasticsearch data is full, so I'm sure it was a couple weeks since the low disk space warning.

I was wondering if anyone had any ideas on how to get this thing running again (unfortunately I am unable to rebuild, migrate or upgrade because of the customer's "requirements"). I thought maybe to delete some indexes but I am not sure if that would break elasticsearch. Any help would be greatly appreciated.

2

Hi @jsutton1 ,

I think that adding some disk space can be a good start, after that you'll certainly be able to start the server and remove index as you plan to do.
Do you have something writing in? an applications or cron task that use it?
Even with more disk space, server may run out of memory (RAM), maybe useful to also check after that.

Hope it help.

3

Thank you @gabriel_tessier for the suggestion. Unfortunately, this is running on a physical box and there is no way to add more disk or memory. (I am pretty new to linux so I'm not sure how else to move ext4 partitions around). This box was supposed to be retired 4 years ago but the customer refused to acknowledge the potential issues. This server is for syslog on some older systems. All logs get forwarded to this server, then NXlog (which caches its different set of disks) forwards them to logstash and qradar.

Can I delete indexes from the partition without elastic running or will that break elastic search when it starts?
I was hesitant to do that because I do not know how elastic would react to missing indexes if they are not removed through the api.

Thanks!