Hi,
I inherited this application when the previous admin left the company and I am not well versed in it. At the moment, I have an issue where I can't log into Kibana and two out of the 4 nodes are down. Trying to log into Kibana with the proper username and password just fails "we couldn't log you in."
Looking at some logs, I see a message "disk usage exceed flood-stage watermark." With that in mind, I believe I need to delete some data. Ideally, I'd like to delete the oldest set of data but I'm not sure how to find this data set or really if this will solve my problem. When I use CURL to view the indices it shows most are yellow and returns data, some are red and show no data relative to their size. To that end, I see a couple of indices I would like to delete but I'm not sure what the identifier to use to delete it and/or if it will have an impact on my issue.
Can some kind soul point me in the direction of if I can select any index to delete and it will help free up the proper space for Kibana to allow log in and/or what else I can do to help my issue?
basically red index is in lost status. may be one or more shard is located in failed node and elastic can't completely see it.
in past I did delete some when I was in tight situation. but it is not safe
I went to /data/elasticsearch/indices/VTtWzKUTTyeS9xC1HPaeZg/0/index
then did listing of files. from files created first
then did
strings _d.cfs for example gives me strings which I knew belongs to which index. took some jugement call and deleted whole dir. got space back and from there kibana worked.
but you have to make sure you stop all ingestion otherwise it will get filled up pretty fast before you do anything.
another route is if you know your approximate index size you can gauge that
]# du -sh indices/* |grep G
18G indices/wkh1xTeNQmOg8bIF4SK66Q ( now i know I only have 10+ gig index only one, and if you confident that you don't need that. delete whole dir.
NOTe: this might delete some dir which can cause your cluster to completely wipe out. do it on your own risk. you have to be pretty good on Linux to do this.
Thank you for your reply. I did a CURL delete on an old index and received an acknowledgment of true; the index in question was 50GB but I'm not seeing that space reclaimed. Neither df nor CURL on the shards shows any change to available space.
From what I read, deleting documents would require a merge to take place to reclaim space but deletion of an index should automatically reclaim but I'm not seeing that. Is there anything I need to do to force this?
Thank you.
I believe I found why the space didn't reclaim; I probably deleted an index from a different shard. I found one on the shard I wanted and I see the space reclaimed and I can log into Kibana again.
I'm still down two nodes (which seem to be because their root partitions filled up). I am working to try and clean up their partitions and see if the nodes come back online.