Delete old data from Static index


#1

Tools:
Elasticsearch: 2.4
Curator: 4.2.3.post1

Index:

green open weblogic 5 1 553689832 0 284.5gb 142.1gb
green open apache 5 1 293808899 0 140.4gb 70.2gb

I would like to retain only last 2 weeks documents from the above two index.
How do I do it?
I tried curator - initially to show the matching index then to delete

Curator Command:

curator_cli
--host internal-elk-mgmt-int-218236766.us-east-1.elb.amazonaws.com
show_indices --filter_list '[{"filtertype":"age","source":"name","direction":"older","timestring":"'%Y.%m.%d'","unit":"days","unit_count":14},{"filtertype":"pattern","kind":"prefix","value":"apache"}]'

I am not exactly sure on how to use this curator - kinda confusing to me
Or if there is any other tools kindly suggest me


(Aaron Mildenstein) #2

Curator is for managing entire indices. It will not delete data from within an index. You need the delete_by_query plugin if you're going to do that.

I recommend, however, using time-series indices for the future so you can just drop indices with Curator, as running a delete_by_query to delete gigabytes of data is a painfully slow operation, by comparison.


#3

Thanks Aaron
After some struggle was able ti use delete_by_query, is there any specific output on the delete action? I am able to see only he delete time status via the HQ plugin

Also, for time-series indices - could you share some insights?
Is there any script available to do it?


(Aaron Mildenstein) #4

How is your data getting into Elasticsearch now? There are a few ways you can do it. One is with the rollover API, which has a 1:1 alias mapped so your data would always be sent to the same index/alias name, and old data is rolled out. Logstash and Beats automatically send data into named time-series indices based on a timestamp field.


#6

I have 4 major index: weblogic, apache, lambda & ec2
The logstash servers feed to elastic-search

Not sure - Correct me if I am wrong, what I thought was I could create index based on date weblogicfeb012017 and alias it to weblogic
Next day create weblogicfeb022017 alias it to weblogic and close the old one

Will rollover satisfy this?


(Aaron Mildenstein) #7

It should, yes.


#8

Great! Will try that

If not rollover do i have any other options?


(Aaron Mildenstein) #9

That depends largely on how you're sending that data right now.


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.