Index filled EC2 completely, manually deleted nodes, no longer able to start Elasticsearch

Elastic Stack Elasticsearch
1

I recently had an issue where I was ingesting data from S3 via Logstash and the S3 Input Plugin (aka the pipeline feature) into my AWS EC2 instance running ELK 7.5.2.

Overnight, the index filled up my entire EC2 storage to the point that I was not able to load Kibana fully (couldn't manage the indices through Kibana because it was stuck loading). So I used the du -sh command to see what was taking up space and it pointed to the "/var/lib/elasticsearch/nodes/0/indices/*" directory.

I looked inside the folders in this directory and the state file seemed to have the name of the index it corresponded to, so I deleted a few of the indices I created to free up space. This indeed freed up massive amounts of space (several GB's) but Elasticsearch could no longer start.

This is the same issue as described here: Elastic search couldnt start after deleting the nodes-HELP! URGENT

The solution was stated as "unfortunately this will have left this node in a broken state. There are no user-serviceable parts inside the data path and you should never make any changes to it yourself.

The only sensible path forwards is to wipe this node. This will allow it to start, and then Elasticsearch will recover the replicas from the other nodes in the cluster."

My only question is how to wipe the node. I was running a single node, single cluster setup so I'm not sure how to do this without essentially deleting ELK off of my EC2.

2

It looks like your data path was /var/lib/elasticsearch so you will need to delete the contents of this directory.

1 Like
3

Perfect, thanks!

How can I configure ELK 7.5.2 so this storage space issue doesn't happen again?

E.g. in the retention policy and do the watermark features work in 7.5?

4

I don't remember any changes in this area since 7.5 although that version is pretty old and long past EOL so I can't be sure. In supported versions Elasticsearch stops accepting indexing when the disk reaches 95% full to avoid completely running out of space.

5

Ok cool, where can I find the documentation on that? Perhaps I can just view the 7.5 version of the docs if available.

6

One other issue I am facing after doing this is that Kibana is stuck on the "Kibana server is not ready yet" phase for several hours now.

Running sudo systemctl status kibana is showing that [security_exception] failed to authenticate user [kibana] and that "license information could not be obtained from Elasticsearch for the data cluster"

7

The latest docs are at Cluster-level shard allocation and routing settings | Elasticsearch Guide [8.11] | Elastic although it looks like they have been reorganised since 7.5. In any case you should upgrade as a matter of urgency, these old versions are not supported.

You will need to ask in the Kibana forum about that, sorry, I don't know how to troubleshoot this sort of thing in Kibana.

1 Like
8

The reason I am stuck using 7.5 is that the way plugins work was changed in a later version of ELK and I'm using the SigmaUI plugin by SOC Prime. If there's a way to port plugins to newer versions of ELK I would gladly do so.

And will do, thanks for all the help so far!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.