Automate deletion of indices in aws elasticcluster

Elastic Stack Logstash
1

I have 3 questions:

  1. how to automate deletion of indices stored in aws_es:
    I am using an aws_elasticsearch to store data where each index is created everyday based on the time stamp they have. I am using -%{+YYYY.MM.dd} at the end of my index name. how can i delete indices which are older than x days and, i want to automate this process.

  2. how does logstash keeps a look at files? I want to perform ondemand indexing where i fetch files in the directory where logstash is running, will those files get indexed as i failed to do so by just creating a file with different timestamp.

  3. The files in my directory are created each hour, and i am creating one index each day, will all the data will go into the same index which is created each day?

2

This is really an elasticsearch question, not a logstash question. I suggest you read up on ILM and post any questions you have about it in the elasticsearch forum rather than this one.

If you are using daily indexes then yes, all the data for one day, based on the UTC timestamp, goes in to one index. That means that for most of us the data for one day in the local timezone is split across two indexes.

3

ILM is not available on AWS ES, so you probably need to use Curator or switch to Elastic Cloud.

1 Like
4

Thanks for the reply christian but as far as i know Curator doesn't works with aws es, here is the description (https://www.elastic.co/guide/en/elasticsearch/client/curator/5.1/faq_aws_iam.html)

5

What version of AWS Elasticsearch are you using? It would have to be a very old version for it to not work with Curator. Amazon patched their versions to include the necessary API calls a long time ago.

6

It's even a recommended tool for AWS ESS:
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/curator.html

But yeah versions matter.

7

Thanks everyone for their reply. I managed to delete the indices using a ruby script.

8

@Shubham_Yadav1, Would mind sharing the ruby script for posterity on this community.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.