We're creating weekly indices and want to rollover whenever there is a new week, using Curator.
However, we see the only "age" based option is 'max_age' which is not exactly what we want to do.
If we set the max_age option to rollover when the index is 7 days old, that is based on the creationdate right? So if i create index1 in the middle of the week, then curator will not rollover the index until the same day next week right?
What we want to do is to let curator determine the age based on the index name, so if we create an index named "logstash-2019.49-00001" curator should look at the name and figure out that the index is for week 49 and as soon as we go over into week 50, curator should rollover.
This is a great question, and the answer is that the problem lies not within Curator but the Rollover API's way of doing things. The Rollover API only provides 3 ways to trigger a rollover: max_age, max_docs, and max_size. As you have correctly identified, max_age is in seconds since index creation, and therefore is not a good fit for what you're doing.
To truly do weekly indices as described, you would have to have Logstash use its date math functionality to simply create a new index every week, and not use the Rollover API. You would then need to use Curator to manage the index life cycle.
More information on the Joda time string representations can be read here, in the date filter plugin documentation. You would perhaps be able to generate what you want in the elasticsearch output with an index directive like:
index => "logstash-%{+yyyy.ww}"
Where y indicates each digit of a 4 digit year, and w indicates each digit of a 2 digit week of year.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.