Time series data, use template to auto-create monthly

Elastic Stack Elasticsearch
1

I have time series data but instead of automatic creation of an index based on the @timestamp value, I want to be able to create it from a value within the document itself, and I'd also like to have the index created monthly rather than daily if possible.

So for example I might have a "my_date" field in the mapping, each time the month rolls over it would parse out "yyyy-mm" or whatever and create a new index "my_index-2016-05" from the specified template (which would roll into an alias "my_index" perhaps).

Is that possible, or am I going to need to provide custom index & alias management for that?

Thanks!

2

Nevermind, I figured it out.. I was simply misunderstanding how templates worked.

3

Since you are asking about @timestamp, I presume you are using logstash? You should ask in the logstash category. From the elasticsearch side, the index name is part of the indexing request. As far as aliases, you will need to manage these yourself, or use something like curator.

4

@ronchalant

Hi Ron, I'm trying to create indices automatically on a monthly basis.

Your input on how you managed to do yours, would help me! Sorry for pulling out an old thread

Thanks.

5

Sorry for the delay, I just saw this.

Basically my initial assumption about how the templates work was incorrect. I was using the marvel template and indices as a reference, and made the false assumption that the format was somehow part of elastic.

It's not. Basically the template property with a trailing * is what is used by Elastic to determine upon implicit index creation if there is a template to use.

so if you create your template with a property like "template": "myindex_*" and then adjust your indexing process so that when it goes to index it generates the index name dynamically, you would have the index be "myindex_201702" or something like that if you want them to be monthly.

I hope (?) that clarifies

1 Like
6

Thank you so much Ron.

I actually rolled out the time-based indices by doing this in my elasticsearch output:

index => "myindex-%{year}-%{month}"

year and month are fields from my database table which actually made my life more easier.

Thank you again :slight_smile: