Remove old data from database

Elastic Stack Logstash
1

Hi,
How can I remove data older from 12 months from database? I have a ELK+Elastiflow to collect logs from some equipments, but I need to do a cleanup of records older than 12 monts.

2

how is your index created. one index per day, per month or everything is in one index?

if you have seperate index then it is easy you can drop them.
if everything is in one index then it is going to be pain

POST index-name/_delete_by_query?conflicts=proceed
{
  "query": {
    "range": {
      "date_field_name": {
        "gte": "02-02-2020",
        "lte": "02-19-2020",
        "format": "MM-dd-yyyy"
      }
    }
  }
}

This will be painfully slow

3

just dropping index from command line

curl -u username:passwd -X DELETE "hostname:9200/index-name?pretty"

4

How can I confirm how and where my indexes was created?

5

I ran curl -XGET -u USER http://localhost:9200/elastiflow-*/_stats?pretty and I see some indexes by date like this:

elastiflow-3.5.2-2020.01.19
elastiflow-3.5.2-2020.01.20
elastiflow-3.5.2-2020.01.21

In this case, if I just run:
curl -u username:passwd -X DELETE "hostname:9200/elastiflow-3.5.2-2020.01.20?pretty"
curl -u username:passwd -X DELETE "hostname:9200/elastiflow-3.5.2-2020.01.21?pretty"

I delete this indexes and save disk space?

6

yes. you can even automate it from bash script

date --date="3 month ago" +%Y.%m.%d (will give you exact format date and you can assign this to variable and remove daily)
2019.12.05

7

Ok. Thank you so much. Save my day.

8

come visit the forum ask question and answer and you will learn in no time.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.