# Rollup data in ES **URL:** \ **Category:** Elasticsearch\ **Created:** [September 23, 2014, 3:07am UTC](https://discuss.elastic.co/t/rollup-data-in-es/19926 "2014-09-23T03:07:40Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [September 23, 2014, 3:07am UTC](https://discuss.elastic.co/t/rollup-data-in-es/19926/1 "2014-09-23T03:07:40Z")
Hi, We are currently utilizing ES for analyzing the last 24 hours of data. The arrival rate of data is of the order of a few hundreds per 10 second interval with each document having a timestamp associated with it. We now need to be able to analyze data over a week and to reduce the amount of space required we plan to retain the 24 hr TTL on the documents but aggregate data into one document per minute to retrieve data that is dated beyond 24 hours and upto 7 days. All fields in the document need to be aggregated. ``` So, ``` 1. Is there any out of the box features that I can use to achieve this kind of roll ups? 2. What is the best approach (preferably a time-tested one if someone has already done this)? 3. Aggregating the data in real time (outside ES) and store the aggregated data into ES 4. Periodically (say once in 30 mins) run aggregation queries and write back responses to ES 5. Periodically (say once in 30 mins) read new documents using time range, aggregate and store back aggregated data in bulk into ES. Maybe use streaming or paged read of documents to aggregate them.... 6. Maybe use a combination of 1 and (2 or 3) so that real time data gets aggregated and data that is delayed (may happen) due to some reason can be updated into the aggregated data using the Update API of ES? Thanks for all the advices, Srinath. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHhx-GJa\_1Qeko20C%3DSaYdWYOJt1EmW-oq8Nj931by4Ab3CDkA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHhx-GJa_1Qeko20C%3DSaYdWYOJt1EmW-oq8Nj931by4Ab3CDkA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [September 23, 2014, 4:17pm UTC](https://discuss.elastic.co/t/rollup-data-in-es/19926/2 "2014-09-23T16:17:41Z")
Hi, We aggregate outside of ES, in memory, and push in bulk. We could still roll up the data stored in ES later on if we wanted to, but reading from ES could get expensive. ## Otis Monitoring \* Alerting \* Anomaly Detection \* Centralized Log Management Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Monday, September 22, 2014 11:08:09 PM UTC-4, Srinath C wrote: > Hi, > We are currently utilizing ES for analyzing the last 24 hours of data. > The arrival rate of data is of the order of a few hundreds per 10 second > interval with each document having a timestamp associated with it. > We now need to be able to analyze data over a week and to reduce the > amount of space required we plan to retain the 24 hr TTL on the documents > but aggregate data into one document per minute to retrieve data that is > dated beyond 24 hours and upto 7 days. All fields in the document need to > be aggregated. > > ``` > So, > > ``` > > 1. Is there any out of the box features that I can use to achieve this > kind of roll ups? > > 2. What is the best approach (preferably a time-tested one if someone has > already done this)? > > 3. Aggregating the data in real time (outside ES) and store the aggregated > data into ES > > 4. Periodically (say once in 30 mins) run aggregation queries and write > back responses to ES > > 5. Periodically (say once in 30 mins) read new documents using time range, > aggregate and store back aggregated data in bulk into ES. Maybe use > streaming or paged read of documents to aggregate them.... > > 6. Maybe use a combination of 1 and (2 or 3) so that real time data gets > aggregated and data that is delayed (may happen) due to some reason can be > updated into the aggregated data using the Update API of ES? > > Thanks for all the advices, > Srinath. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/381cc014-cf85-4235-9552-af33d9629c1d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/381cc014-cf85-4235-9552-af33d9629c1d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [September 24, 2014, 1:39pm UTC](https://discuss.elastic.co/t/rollup-data-in-es/19926/3 "2014-09-24T13:39:40Z")
Otis, Thanks for the reply. On Tuesday, 23 September 2014 21:47:42 UTC+5:30, Otis Gospodnetic wrote: > Hi, > > We aggregate outside of ES, in memory, and push in bulk. We could still > roll up the data stored in ES later on if we wanted to, but reading from ES > could get expensive. > > ## Otis > > Monitoring \* Alerting \* Anomaly Detection \* Centralized Log Management > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > On Monday, September 22, 2014 11:08:09 PM UTC-4, Srinath C wrote: > > > Hi, > > We are currently utilizing ES for analyzing the last 24 hours of > > data. The arrival rate of data is of the order of a few hundreds per 10 > > second interval with each document having a timestamp associated with it. > > We now need to be able to analyze data over a week and to reduce the > > amount of space required we plan to retain the 24 hr TTL on the documents > > but aggregate data into one document per minute to retrieve data that is > > dated beyond 24 hours and upto 7 days. All fields in the document need to > > be aggregated. > > > > ``` > > So, > > > > ``` > > > > 1. Is there any out of the box features that I can use to achieve this > > kind of roll ups? > > > > 2. What is the best approach (preferably a time-tested one if someone has > > already done this)? > > > > 3. Aggregating the data in real time (outside ES) and store the > > aggregated data into ES > > > > 4. Periodically (say once in 30 mins) run aggregation queries and write > > back responses to ES > > > > 5. Periodically (say once in 30 mins) read new documents using time > > range, aggregate and store back aggregated data in bulk into ES. Maybe use > > streaming or paged read of documents to aggregate them.... > > > > 6. Maybe use a combination of 1 and (2 or 3) so that real time data gets > > aggregated and data that is delayed (may happen) due to some reason can be > > updated into the aggregated data using the Update API of ES? > > > > Thanks for all the advices, > > Srinath. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/44224334-abcb-4794-893e-d2f4f0f199ba%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/44224334-abcb-4794-893e-d2f4f0f199ba%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:00am UTC](https://discuss.elastic.co/t/rollup-data-in-es/19926/4 "2017-07-06T01:00:14Z")