# Can an Elasticsearch rollup job dynamically create indexes like Logstash does? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 21, 2018, 2:01am UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268 "2018-08-21T02:01:14Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![stuart-j-campbell](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@stuart-j-campbell](https://discuss.elastic.co/u/stuart-j-campbell)\ **Post date:** [August 21, 2018, 2:01am UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/1 "2018-08-21T02:01:15Z")
I am currently testing out the new rollup APIs in Elasticsearch 6.3 and am wondering if there is any way to configure the rollup job to dynamically create an index based on timestamp like Logstash does when ingesting data? The use case is to try and roll up large amounts of time series network performance reporting data and I'm worried that even an hourly rollup will create a huge index to manage so am looking to split it to have one index for each day's hourly rollup. Current rollup job config: ``` { "index_pattern": "dxs-raw-*", "rollup_index": "dxs-hourly-%{+YYYY.MM.dd}", "cron": "* */15 * * * ?", "page_size": 1000, "groups": { "date_histogram": { "field": "@timestamp", "interval": "1h", "delay": "12h" }, "terms": { "fields": ["ci_id.keyword", "client_id.keyword", "element_name.keyword", "measurement.keyword", "source_management_platform.keyword", "unit.keyword"] } }, "metrics": [ { "field": "value", "metrics": ["min", "max", "avg"] } ] } ``` Error seen when PUTting job via Kibana DevTools console: ``` { "error": { "root_cause": [ { "type": "invalid_index_name_exception", "reason": "Invalid index name [dxs-hourly-%{+YYYY.MM.dd}], must be lowercase", "index_uuid": "_na_", "index": "dxs-hourly-%{+YYYY.MM.dd}" } ], "type": "runtime_exception", "reason": "runtime_exception: Could not create index for rollup job [dxs-hourly]", "caused_by": { "type": "invalid_index_name_exception", "reason": "Invalid index name [dxs-hourly-%{+YYYY.MM.dd}], must be lowercase", "index_uuid": "_na_", "index": "dxs-hourly-%{+YYYY.MM.dd}" } }, "status": 500 } ``` ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [August 21, 2018, 9:58pm UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/2 "2018-08-21T21:58:10Z")
At the moment, no... there's no way to do that. Right now we're sorta waiting on the [Index Lifecycle Mangement](https://github.com/elastic/elasticsearch/issues/29823) feature to become available, since that will allow this sort of management a lot easier than baking parts of the logic into the Rollup feature. But that's still a work in progress. It _might_ be possible to use the [Rollover API](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html) today, but it's entirely untested so I'm not sure. E.g. create a write alias, setup some rollover rules, then point the Rollup config at the alias. The tricky bit is that Rollup's disaster recover mechanism (like if a node goes down) is to simply backtrack to the last checkpoint and overwrite any documents that were written after the checkpoint. So you'd want to make sure that doesn't happen around a rollover event. A procedure like: 1. Stop rollup job, wait for it to finish and checkpoint 2. Hit Rollover api to check conditions 3. Restart rollup job The only other alternative is to pin the job to a specific source index and create lots of jobs, each representing that single index it rolled up. Note: Rollup right now doesn't allow searching across multiple rollup indices (technical limitations internally), so you'd be stuck searching _just_ the single hourly rollup index which is highly non-ideal 😕 How many rollup documents is your job generating for each hour, or other timeframe? ---
**Author:** ![stuart-j-campbell](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@stuart-j-campbell](https://discuss.elastic.co/u/stuart-j-campbell)\ **Post date:** [August 22, 2018, 12:10am UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/3 "2018-08-22T00:10:04Z")
Thanks Zachary, I will try the alias idea out and report back 🙂. The conservative estimate we have is ~1 million metrics if it was a single hourly index. Given our document size this equates to about 160GB of data. If what I'm building manages to get high adoption that number could be 3-4x higher. Our "raw" data is stored in daily indexes of ~72,000,000 documents in size (~12GB). We are essentially rolling up progressively so once we get past hourly the numbers (using the same numbers as above) would look like: Daily = 90,000,000 / 13.41GB Weekly = ~20,000,000 / 2.87GB Monthly = 6,000,000 / 0.89GB Quarterly = 3,000,000 / 0.45GB Yearly = 1,000,000 / 0.15GB ---
**Author:** ![stuart-j-campbell](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@stuart-j-campbell](https://discuss.elastic.co/u/stuart-j-campbell)\ **Post date:** [August 22, 2018, 1:30am UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/4 "2018-08-22T01:30:32Z")
Hey Zachary, did try the alias and rollover idea out and still no luck I'm afraid. Here is what I tried as a test to rollup every ten minutes into hourly rollover indexes: Created a new index with an alias: ``` PUT /%3Cdxs-rolled-hourly-%7Bnow%2FH-1H%7BYYYY.MM.dd.HH-%7D%7D%3E { "aliases": { "dxs-rolled-hourly": {} } } ``` Created a new rollup job targeting the alias: ``` PUT _xpack/rollup/job/dxs-hourly { "index_pattern": "dxs-raw-*", "rollup_index": "dxs-rolled-hourly", "cron": "*/5 * * * * ?", "page_size": 1000, "groups": { "date_histogram": { "field": "@timestamp", "interval": "10m", "delay": "30m" }, "terms": { "fields": ["ci_id.keyword", "client_id.keyword", "element_name.keyword", "measurement.keyword", "source_management_platform.keyword", "unit.keyword"] } }, "metrics": [ { "field": "value", "metrics": ["min", "max", "avg"] } ] } ``` Received the following error: ``` { "error": { "root_cause": [ { "type": "invalid_index_name_exception", "reason": "Invalid index name [dxs-rolled-hourly], already exists as alias", "index_uuid": "_na_", "index": "dxs-rolled-hourly" } ], "type": "runtime_exception", "reason": "runtime_exception: Could not create index for rollup job [dxs-hourly]", "caused_by": { "type": "invalid_index_name_exception", "reason": "Invalid index name [dxs-rolled-hourly], already exists as alias", "index_uuid": "_na_", "index": "dxs-rolled-hourly" } }, "status": 500 } ``` So to me looks like the index is being created rather than referenced (thus the error when I used the same name as the alias). Since it looks like this isn't possible today what would you recommend as the approach to get this raised as a potential enhancement? ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [August 22, 2018, 2:40pm UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/5 "2018-08-22T14:40:21Z")
Ah, right, I forgot about that. When you create a job, it first tries to create the destination rollup index. If it exists, it updates the existing rollup metadata to include a new job. But if it's an alias, it'll just throw an exception like you saw. I think that should be fixable. I'll open a ticket in a few minutes with this enhancement request, there are a few routes we could take (fix for aliases, support for rollover internally, ILM, etc). > The conservative estimate we have is ~1 million metrics if it was a single hourly index. Given our document size this equates to about 160GB of data. If what I'm building manages to get high adoption that number could be 3-4x higher. Our "raw" data is stored in daily indexes of ~72,000,000 documents in size (~12GB). Sorry, a bit confused here. Are you saying one hour of rolled up data generates 160gb, but the raw data of 72m docs is only 12gb? Ditto for some of the other numbers, like 90m/13.41GB in rollup compared to 72m raw? I think I may just be misunderstanding something here, sorry! ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [August 22, 2018, 4:38pm UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/6 "2018-08-22T16:38:12Z")
FYI, I opened an issue to track this: [https://github.com/elastic/elasticsearch/issues/33065](https://github.com/elastic/elasticsearch/issues/33065) ---
**Author:** ![stuart-j-campbell](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@stuart-j-campbell](https://discuss.elastic.co/u/stuart-j-campbell)\ **Post date:** [August 23, 2018, 12:49am UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/7 "2018-08-23T00:49:43Z")
Thanks Zachary! Sorry, the way I phrased the sizing isn't very clear, here is what we are trying to do: 1. Collect performance metrics for devices every 5 minutes and store them in an index for each day. Estimate is a document size of about 160 bytes, 50 metrics per device, collected every 5 minutes, for 5000 devices. That gets to our first figure of 72,000,000 documents per day which would be stored in a daily index which comes out at just under 12GB **for each daily index**. We are planning to retain each daily index for 90 days which ends up being just under 1TB of data. 2. As this is a significant amount of data we then wanted to create rollups for hourly, daily, weekly, monthly, quarterly, and yearly averages for all metrics. Ideally in manageable index sizes so we were hoping to have a daily index for hourly rollups, a weekly index for daily rollups, etc. If we had to store all the hourly rollup documents for the planned six month retention period in a single index that index would be ~160GB in size. Does that help clear it up? ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [August 23, 2018, 12:33pm UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/8 "2018-08-23T12:33:04Z")
Gotcha, that makes sense. Thanks for laying it all out... it helps me get a clearer idea of the scale we're talking. Some thoughts in no particular order: - We've been actively discussing how to do some kind of partitioning scheme, will post updates to that ticket as we work on it. - Is that 160gb size for the six-month period extrapolated from smaller Rollup tests, or just theoretical size based on the raw data? I'm asking because the Rollup documents have a different structure than the original, which has different compression characteristics (even ignoring the rollup effect, just due to data layout/format/density). Curious to see what kind of compression values you're getting over the raw data for a specific period of time. - I know you said you expect the project to grow, but for now I think 160gb is right on the edge of OK. With five shards, a 160gb index comes out to ~30gb/shard, which is perfectly reasonable. Logging/metrics use-cases can easily push 50-100gb/shard. But I agree that much more volume and some kind of rollover situation is probably in order. - What data type is your original data? All `long` and `float`/`double`? Or do you also use the newer `half_float` and `scaled_float`? Asking because right now Rollup stores all metrics as `double`, which are quite bulky... so if we enable `float`/`half_float`/`scaled_float` it would compress better. Thanks for all the info! Being new and experimental, real user data like this is super helpful for how we should tweak/optimize/enhance Rollup. 🙂 ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [August 23, 2018, 7:53pm UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/9 "2018-08-23T19:53:47Z")
Also, an unrelated note but since you're actively testing Rollup, I thought I should mention it. We [fixed a fairly serious issue](https://github.com/elastic/elasticsearch/issues/32372) with the document IDs used by Rollup. I'd suggest upgrading to 6.4 before any production data is permanently stored 🙂 ---
**Author:** ![stuart-j-campbell](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@stuart-j-campbell](https://discuss.elastic.co/u/stuart-j-campbell)\ **Post date:** [August 27, 2018, 3:11am UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/10 "2018-08-27T03:11:08Z")
Thanks for all the help Zach, to answer your questions: 1. The 160GB was a theoretical size based on the existing raw data. I'm running a rollup job in my QA environment at the moment (on 6.4 🙂) and will let you know how it goes. 2. Data is currently float, I hadn't seen the new data types. We could definitely go to half\_float and potentially scaled float. I will chat to the team about that to save some space as well. Having those types in the rollup would be great. ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [August 27, 2018, 7:10pm UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/11 "2018-08-27T19:10:36Z")
Great, thanks! Looking forward to what your tests show 🙂 ---
**Author:** ![stuart-j-campbell](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@stuart-j-campbell](https://discuss.elastic.co/u/stuart-j-campbell)\ **Post date:** [August 29, 2018, 5:03am UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/12 "2018-08-29T05:03:43Z")
Ok got it working (can I suggest a docs update that mentions Rollup doesn't play nicely with index templates? 🙂). What I end up with is a rollup index with a document size of ~210 bytes which if we use the projections is actually a lot bigger than I was predicting at ~220GB so I definitely think the different data types would help with the compression. But we are pretty happy with the results. So in the end we are looking to go with a single rollup index for each of our rollup types spilt into 10 shards. ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [August 30, 2018, 1:50pm UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/13 "2018-08-30T13:50:11Z")
> [@stuart-j-campbell](#): > > can I suggest a docs update that mentions Rollup doesn't play nicely with index templates? Sure... what bit was the difficulty here? Broadly matching patterns being applied to rollup index on accident or something similar? > What I end up with is a rollup index with a document size of ~210 bytes which if we use the projections is actually a lot bigger than I was predicting at ~220GB so I definitely think the different data types would help with the compression. > > But we are pretty happy with the results. So in the end we are looking to go with a single rollup index for each of our rollup types spilt into 10 shards. Sounds good, thanks for the update! I think adding support for `scaled_floats` will help a bunch, since they are _vastly_ smaller than `double`. Even `float`/`half-float` would offer a significant savings. Just a guess, but I think most folks will be ok with the precision trade off here given it's rolled data. We're also looking into what we can turn off, e.g. do we really need to index the values for search, or is it ok to just generate doc\_values. Things like that should improve size over time as we implement it. ---
**Author:** ![stuart-j-campbell](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@stuart-j-campbell](https://discuss.elastic.co/u/stuart-j-campbell)\ **Post date:** [September 3, 2018, 12:10am UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/14 "2018-09-03T00:10:27Z")
Hey Zach, yes that was the exact problem with the index templates. It surfaced strangely though in the sense that the rollup job created successfully but just didn't populate with data until I removed the index template and recreated the job. That all sounds excellent I can definitely voice at least one agreement on the tradeoff between precision and size 🙂. Thanks again for all your help on this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 1, 2018, 12:10am UTC](https://discuss.elastic.co/t/can-an-elasticsearch-rollup-job-dynamically-create-indexes-like-logstash-does/145268/15 "2018-10-01T00:10:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.