Hi, how can i release storage from my Elasticsearch server, i receive logs from 6 other servers from filebeat and logstash, and have an index that contains 190GB and the server is with only 5% of free storage, Although, I used delete by query, but this maybe is ineficient, What is the other ways?
I need to delete data oldest than 80 days.
The typical approach to cope with "data retention" is:
- to write data in time-based indices (e.g. using the date math support)
- use Index Lifecycle Management if your cluster is on version 6.7+
In this way, you can delete old data just dropping the old indices.
Logstash & Filebeat support those methods natively.
In your case you have this index which has a size of 190GB and little place for maneuvering.
The only option with current hardware and without outages is to proceed with delete by query.
It is inefficient because the data will be actually removed from disk only after a merge which will occur in background if the number of delete documents in a segment is more than a certain threshold.
If you do not have updates over the documents (you always index), I would:
- Start writing to a new time based index.
- Delete by query the old data
- Once it's done, forcemerge the index and set it to read only. This will expunge the deleted documents.
If the data to be kept is less than then half of data you have in the index, another approach would be to reindex to a new index only the documents you want to keep.
Once done, you could delete the old index. But you cannot apply this strategy because you do not have enough disk space to perform this operation.
Depending on what you can do (e.g. add a new node or increase the disk size), there might be other strategies.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.