Context:
I am working on building out an aging scheme for time based indexes for log data. I know about closing/opening, snapshot/restore, flushing and deleting, however each of these methods functions off the presumption that you are more concerned with the Elasticsearch index entities themselves, and not the powerful data control and query results that Elasticsearch offers. When it comes to long-term statistics there comes a point when I don't need elastic anymore, I just need some specific aggregation and query results to get the "summery" of historical events, currently stored in indexes.
I am trying to figure out how to automatically generate and store statistical data (generated as query results) from existing indexes (populated by docs that are individual traffic and error logs) and store this information in a new index before deleting the queried indexes, thus retaining the "meat" of the data (which is the only important part after a period of time for gleaning historical use patterns: a long term issue) while thinning out the extraneous information that is only needed to resolve bugs and address traffic issues: a problem of the now.
Update: I also have looked into Curator, but it does not seem to have been built into versions beyond 5.2. Is it still supported in 5.6? Additionally I have looked at just writing a cron job that makes my searches, stores the results, and PUTs the results in a new index, then deletes the same indexes searched, all via the RESTful API from the server my nodes are running on. This would work, but I am interested if this is possible inside Elasticserach.
Question 1:
Is there a way to internally automate queries/searches/aggregations at specified times and store the results of this search as a document in a new index?
Question 2:
Is there a way to automate the deletion of indexes after a specified age?