we're looking to use the rollups functionality (on ES 7.4) in order the reduce size of the data for our search queries.
We have a monthly lifecycle strategy for our data: each month data is stored in a separated index with a naming
[index_name]_[year]-[month]. We usually execute search queries using a group of these indices, and want to replace this searches using a rollup index plus 1 or 2 indexes.
In order to do this, we are trying to create a rollup job using as "index_pattern":
[index_name]_*. The documents for existent indices are processed by the job, but no document included on a new index created after the job.
If we create another job after the new index is created, these new documents are processed.
We also wanted to use data filtered using a filter alias, created with similar naming pattern on each index, and using a similar value for "index_pattern", but same issue.
Is it possible for a rollup job configured with a "index_pattern" to process data for future indices to be created (that apply in the same "index_pattern")? Or only it processed new data for the existent indices when the job was created?
Any suggestion to what we are trying to accomplish? Create for each new index a new rollup job using same "rollup_index"? Or there could be some issue to use same rollup index for different rollup jobs?