my team and I are looking for a solution to keep the size of indices below a certain threshold and we are looking at the Rollover API with interest. We are already using Curator for data retention purposes and thought we could add rollover actions there.
One problem we have, though, is with dynamic index names.
We are using Filebeat to ship logs directly to Elasticsearch from tens of different applications running on cloud infrastructure. Each application has its own index name, with monthly suffixes, which simplifies the data retention task.
The question is how do we tell Filebeat that, when creating a new index at the start of each month, it should really create an alias + an initial index that can be rolled over when the size condition matches?
The only solution we have found is to create aliases upfront and point Filebeat to such aliases, but this would need to be done every month and it's not a clean solution.
Is there something we are missing?
Also, how do we create a generic rollover action in Curator that matches all possible indices?
Thanks for any help