I am using Amazon ES, can someone tell me what is the ideal way of automatically deleting the old indexes from the cluster considering Amazon ES does not provide the ILM features ?
I have 25 GB of daily index and I want to keep the data for longer period of time for about 6 months. I have read somewhere that data deletion is not a recommended approach as it involves lot of I/O operation. Then what is the way around if I want to maintain small indexes with less data, deleting the old records ?
Also will there be any impact on the memory if the no. of shards increases while I am deleting the documents older than 30 days daily?
Also if you can answer this query mentioned above
"I have 25 GB of daily index and I want to keep the data for longer period of time for about 6 months. I have read somewhere that data deletion is not a recommended approach as it involves lot of I/O operation. Then what is the way around if I want to maintain small indexes with less data, deleting the old records ?"
I would not recommend that. I am using Elastic Cloud and compared to ES running directly on EC2 it is far less reliable, far more expensive. I have to use curator in addition to ILM. Also, ILM shown strange behavior, which nobody was able to explain. For production use, Elastic Cloud and ILM is far less reliable choice than self-hosted or AWS managed ES with curator.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.