# Scheduled query service **URL:** \ **Category:** Elasticsearch\ **Created:** [February 26, 2019, 9:44pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075 "2019-02-26T21:44:11Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [February 26, 2019, 9:44pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/1 "2019-02-26T21:44:11Z")
Hi, I'm looking for an existing service/application which would execute scheduled queries defined by me and insert the result to an ES index. My goal is to aggregate data to be able to extend key metrics retention time. This way I'd be able to tell trends over a longer period without having granular data which costs a lot of disk space (not to mention replicas and snapshots...). Is there such service or do I need to write an app myself? Thanks! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 26, 2019, 9:44pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/2 "2019-02-26T21:44:57Z")
Why not just use the rollup API? ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [February 27, 2019, 11:20am UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/3 "2019-02-27T11:20:46Z")
It's not clear to me that which components of the X-pack isn't available with free subscription so I usually don't even open anything related to X-pack since I can't afford that. Thank you! I start to read and experiment with it right away 🙂 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 27, 2019, 11:46am UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/4 "2019-02-27T11:46:48Z")
You can see that on that page: [https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions) For example: ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/3/23d5cf531421619820644f7a94c1a98eda9226ff.jpeg) ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [February 27, 2019, 4:12pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/5 "2019-02-27T16:12:54Z")
Thanks, I'll bookmark that page. ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [February 27, 2019, 4:33pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/6 "2019-02-27T16:33:58Z")
How likely will this (or any other with the warning message) be removed? And if it will be removed, usually how long is the deprecation period? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 27, 2019, 4:47pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/7 "2019-02-27T16:47:12Z")
Sorry? Which message? ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [February 28, 2019, 10:39am UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/8 "2019-02-28T10:39:01Z")
Sorry, I meant to include it: ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/7/3730d10d495b645b6f205ee49fb7cd1ac81d7d47.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 28, 2019, 1:36pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/9 "2019-02-28T13:36:47Z")
This message does not mean that we are going to remove the feature. When we want to remove something we are deprecating APIs which is not the case here. This message says basically that: - If we find out that it's a dangerous feature, think about data integrity or things like that, we might say "that was a bad idea, sorry." - but most of the time (if not always) it just means that the API might be subject to change and that at this step we can not guarantee the compatibility between versions of this API. When stable/mature enough (ie we do have lot of positive feedbacks or not so many issues with the API), we can remove the message. I hope this makes sense. ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [February 28, 2019, 4:18pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/10 "2019-02-28T16:18:14Z")
Thanks, that's good to know! I think the Rollup API will be able to solve the majority of the problems I face but there are things it lacks. Or more precisely it isn't in the scope of the API. Let's say I have a million document every day and I'd like to see how is the TOP 10 of XY term changing over time. Even if it isn't a high-cardinality field I wouldn't want to create 3000 groups just to query those again. Sometimes I need to know how many documents I had in a given time period and how many had the field "foo" in it, or the field "bar", etc. Something like: _Document\_total\_count_: 1,000,000 _foo\_exists_: 6,782 _bar\_exists_: 442,332 So I still need a way to store the result of a query which is executed periodically. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 28, 2019, 4:34pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/11 "2019-02-28T16:34:33Z")
May be @polyfractal has an idea? ---
**Author:** ![MarkusT](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/markust/32/30037_2.png) [@MarkusT](https://discuss.elastic.co/u/MarkusT)\ **Post date:** [March 1, 2019, 8:21am UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/12 "2019-03-01T08:21:12Z")
Maybe you could try an exists filter aggregation, like ``` "aggs": { "results_with_field": { "filter": { "exists": { "field": "name" } } } } ``` and then get the cardinality. ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [March 1, 2019, 10:10am UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/13 "2019-03-01T10:10:35Z")
Within the Rollup API? ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [March 1, 2019, 4:35pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/14 "2019-03-01T16:35:28Z")
Sorry for the delayed response everyone 🙂 Yeah, I'm not sure Rollup is a perfect fit here. If you only want the top 10 (or whatever), Rollup will be doing a lot more work since it has to exhaustively save all the groups. It may be a trivial amount of extra work depending on the size of the data, so you could experiment and see. But the design goal of Rollup is to take all your existing data and "downsample" it to a larger granularity, which means you'll be saving more than just the top `n` values. A simpler way would be Watcher/Alerting (define a watch on a scheduled basis which includes the top n agg, save the result), but that's in Gold. 😕 If Rollup doesn't work for you, I think you'll probably have to write a script that executes your query and transforms it into a document, then just schedule it with a cron or something. ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [March 1, 2019, 4:55pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/15 "2019-03-01T16:55:04Z")
Thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 29, 2019, 4:55pm UTC](https://discuss.elastic.co/t/scheduled-query-service/170075/16 "2019-03-29T16:55:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.