Delete old data from Static index

Elastic Stack Elasticsearch
1

Tools:
Elasticsearch: 2.4
Curator: 4.2.3.post1

Index:

green open weblogic 5 1 553689832 0 284.5gb 142.1gb
green open apache 5 1 293808899 0 140.4gb 70.2gb

I would like to retain only last 2 weeks documents from the above two index.
How do I do it?
I tried curator - initially to show the matching index then to delete

Curator Command:

curator_cli
--host internal-elk-mgmt-int-218236766.us-east-1.elb.amazonaws.com
show_indices --filter_list '[{"filtertype":"age","source":"name","direction":"older","timestring":"'%Y.%m.%d'","unit":"days","unit_count":14},{"filtertype":"pattern","kind":"prefix","value":"apache"}]'

I am not exactly sure on how to use this curator - kinda confusing to me
Or if there is any other tools kindly suggest me

2

Curator is for managing entire indices. It will not delete data from within an index. You need the delete_by_query plugin if you're going to do that.

I recommend, however, using time-series indices for the future so you can just drop indices with Curator, as running a delete_by_query to delete gigabytes of data is a painfully slow operation, by comparison.

3 Likes
3

Thanks Aaron
After some struggle was able ti use delete_by_query, is there any specific output on the delete action? I am able to see only he delete time status via the HQ plugin

Also, for time-series indices - could you share some insights?
Is there any script available to do it?

4

How is your data getting into Elasticsearch now? There are a few ways you can do it. One is with the rollover API, which has a 1:1 alias mapped so your data would always be sent to the same index/alias name, and old data is rolled out. Logstash and Beats automatically send data into named time-series indices based on a timestamp field.

6

I have 4 major index: weblogic, apache, lambda & ec2
The logstash servers feed to elastic-search

Not sure - Correct me if I am wrong, what I thought was I could create index based on date weblogicfeb012017 and alias it to weblogic
Next day create weblogicfeb022017 alias it to weblogic and close the old one

Will rollover satisfy this?

7

It should, yes.

8

Great! Will try that

If not rollover do i have any other options?

9

That depends largely on how you're sending that data right now.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.