Elasticsearch housekeeping

Hi,

I am using Amazon ES, can someone tell me what is the ideal way of automatically deleting the old indexes from the cluster considering Amazon ES does not provide the ILM features ?

I have 25 GB of daily index and I want to keep the data for longer period of time for about 6 months. I have read somewhere that data deletion is not a recommended approach as it involves lot of I/O operation. Then what is the way around if I want to maintain small indexes with less data, deleting the old records ?

Also will there be any impact on the memory if the no. of shards increases while I am deleting the documents older than 30 days daily?

Elasticsearch Curator should work for you.

You could always upgrade to our Elasticsearch Service as well :slight_smile:

3 Likes

HI @warkolm

I think curator will not work in Amazon ES.

Also if you can answer this query mentioned above
"I have 25 GB of daily index and I want to keep the data for longer period of time for about 6 months. I have read somewhere that data deletion is not a recommended approach as it involves lot of I/O operation. Then what is the way around if I want to maintain small indexes with less data, deleting the old records ?"

Curator works just fine with AWS Elasticsearch, so long as you are using AWS Elasticsearch version 5.4 or higher.

1 Like

I would not recommend that. I am using Elastic Cloud and compared to ES running directly on EC2 it is far less reliable, far more expensive. I have to use curator in addition to ILM. Also, ILM shown strange behavior, which nobody was able to explain. For production use, Elastic Cloud and ILM is far less reliable choice than self-hosted or AWS managed ES with curator.

Welcome to the community :slight_smile:

Are there other threads that you've posted under a different account with these issues that we might be able to take a look at?

Hi,

Have a look at this:

1 Like