# Index filled EC2 completely, manually deleted nodes, no longer able to start Elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [March 21, 2022, 3:45pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221 "2022-03-21T15:45:10Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![pritster5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pritster5/32/103311_2.png) [@pritster5](https://discuss.elastic.co/u/pritster5)\ **Post date:** [March 21, 2022, 3:45pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221/1 "2022-03-21T15:45:10Z")
I recently had an issue where I was ingesting data from S3 via Logstash and the S3 Input Plugin (aka the pipeline feature) into my AWS EC2 instance running ELK 7.5.2. Overnight, the index filled up my entire EC2 storage to the point that I was not able to load Kibana fully (couldn't manage the indices through Kibana because it was stuck loading). So I used the `du -sh` command to see what was taking up space and it pointed to the "`/var/lib/elasticsearch/nodes/0/indices/*`" directory. I looked inside the folders in this directory and the state file seemed to have the name of the index it corresponded to, so I deleted a few of the indices I created to free up space. This indeed freed up massive amounts of space (several GB's) but Elasticsearch could no longer start. This is the same issue as described here: [Elastic search couldnt start after deleting the nodes-HELP! URGENT](https://discuss.elastic.co/t/elastic-search-couldnt-start-after-deleting-the-nodes-help-urgent/203741) The solution was stated as "unfortunately this will have left this node in a broken state. There are no user-serviceable parts inside the data path and you should never make any changes to it yourself. The only sensible path forwards is to wipe this node. This will allow it to start, and then Elasticsearch will recover the replicas from the other nodes in the cluster." **My only question is how to wipe the node**. I was running a single node, single cluster setup so I'm not sure how to do this without essentially deleting ELK off of my EC2. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 21, 2022, 4:02pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221/2 "2022-03-21T16:02:08Z")
> [@pritster5](#): > > **My only question is how to wipe the node**. It looks like your data path was `/var/lib/elasticsearch` so you will need to delete the contents of this directory. ---
**Author:** ![pritster5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pritster5/32/103311_2.png) [@pritster5](https://discuss.elastic.co/u/pritster5)\ **Post date:** [March 21, 2022, 4:29pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221/3 "2022-03-21T16:29:27Z")
Perfect, thanks! How can I configure ELK 7.5.2 so this storage space issue doesn't happen again? E.g. in the retention policy and do the watermark features work in 7.5? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 21, 2022, 5:05pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221/4 "2022-03-21T17:05:59Z")
I don't remember any changes in this area since 7.5 although that version is pretty old and long past [EOL](https://www.elastic.co/support/eol) so I can't be sure. In supported versions Elasticsearch stops accepting indexing when the disk reaches 95% full to avoid completely running out of space. ---
**Author:** ![pritster5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pritster5/32/103311_2.png) [@pritster5](https://discuss.elastic.co/u/pritster5)\ **Post date:** [March 21, 2022, 6:26pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221/5 "2022-03-21T18:26:03Z")
Ok cool, where can I find the documentation on that? Perhaps I can just view the 7.5 version of the docs if available. ---
**Author:** ![pritster5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pritster5/32/103311_2.png) [@pritster5](https://discuss.elastic.co/u/pritster5)\ **Post date:** [March 21, 2022, 6:33pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221/6 "2022-03-21T18:33:31Z")
One other issue I am facing after doing this is that Kibana is stuck on the "Kibana server is not ready yet" phase for several hours now. Running `sudo systemctl status kibana` is showing that `[security_exception] failed to authenticate user [kibana]` and that "license information could not be obtained from Elasticsearch for the data cluster" ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 21, 2022, 9:30pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221/7 "2022-03-21T21:30:19Z")
> [@pritster5](#): > > Ok cool, where can I find the documentation on that? The latest docs are at [Cluster-level shard allocation and routing settings | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation) although it looks like they have been reorganised since 7.5. In any case you should upgrade as a matter of urgency, these old versions are not supported. > [@pritster5](#): > > One other issue I am facing after doing this is that Kibana is stuck on the "Kibana server is not ready yet" phase for several hours now. You will need to ask in the Kibana forum about that, sorry, I don't know how to troubleshoot this sort of thing in Kibana. ---
**Author:** ![pritster5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pritster5/32/103311_2.png) [@pritster5](https://discuss.elastic.co/u/pritster5)\ **Post date:** [March 22, 2022, 2:18pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221/8 "2022-03-22T14:18:02Z")
The reason I am stuck using 7.5 is that the way plugins work was changed in a later version of ELK and I'm using the SigmaUI plugin by SOC Prime. If there's a way to port plugins to newer versions of ELK I would gladly do so. And will do, thanks for all the help so far! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 19, 2022, 2:18pm UTC](https://discuss.elastic.co/t/index-filled-ec2-completely-manually-deleted-nodes-no-longer-able-to-start-elasticsearch/300221/9 "2022-04-19T14:18:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.