# Aggregations large data in real time in Elasticsearch. What solution is the best? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 5, 2020, 2:10am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188 "2020-03-05T02:10:23Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![ndhcoder](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndhcoder/32/63850_2.png) [@ndhcoder](https://discuss.elastic.co/u/ndhcoder)\ **Post date:** [March 5, 2020, 2:10am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/1 "2020-03-05T02:10:23Z")
Hi all, I have a question ? With a system have about a billion logs each days, How can i get all my aggregations data like "count request error per host", "count request per days, per hours, ..", in time series in real time, that mean when i request aggregations, the query can consume old results before to calculating new result for best performance, how we do that in Elasticsearch .. or exist any way better ? Thanks all. Sorry about my bad english. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 5, 2020, 7:34pm UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/2 "2020-03-05T19:34:20Z")
The basic answer is that Elasticsearch calculates all of that at request time. It doesn't run these sorts of calculations on a scheduled and then store the results for any requests that are made. So any request you make will be a real time one. ---
**Author:** ![ndhcoder](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndhcoder/32/63850_2.png) [@ndhcoder](https://discuss.elastic.co/u/ndhcoder)\ **Post date:** [March 6, 2020, 7:27am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/3 "2020-03-06T07:27:47Z")
Hi, Thanks for reply. What concept of Elasticsearch that i use to implement it ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 6, 2020, 7:37am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/4 "2020-03-06T07:37:48Z")
It sounds like you might be interested in the roll up api. I believe there are videos and. Log posts about it, but do not have links handy at the moment. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 6, 2020, 7:44am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/5 "2020-03-06T07:44:12Z")
I think you need to look at aggregations. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html) ---
**Author:** ![ndhcoder](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndhcoder/32/63850_2.png) [@ndhcoder](https://discuss.elastic.co/u/ndhcoder)\ **Post date:** [March 6, 2020, 9:07am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/6 "2020-03-06T09:07:18Z")
Hi, Thanks for suggestion, but i saw the search aggregations, that calculate again each time we searching, so performance seem bad for large data set, i need tracking realtime ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 6, 2020, 9:07am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/7 "2020-03-06T09:07:39Z")
> so performance seem bad for large data set And did you try it? ---
**Author:** ![ndhcoder](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndhcoder/32/63850_2.png) [@ndhcoder](https://discuss.elastic.co/u/ndhcoder)\ **Post date:** [March 6, 2020, 9:10am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/8 "2020-03-06T09:10:00Z")
No, I read the document, that don't say about any mechanism for cache old results, when my data set is billion document each day, when multi user query for tracking request in realtime. I think it is a bad idea in this case. ---
**Author:** ![ndhcoder](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndhcoder/32/63850_2.png) [@ndhcoder](https://discuss.elastic.co/u/ndhcoder)\ **Post date:** [March 6, 2020, 9:11am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/9 "2020-03-06T09:11:25Z")
The results need to be updated per seconds ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 6, 2020, 10:26am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/10 "2020-03-06T10:26:52Z")
> [@ndhcoder](#): > > I think it is a bad idea in this case. I don't think it is. I'd really recommend that you try before saying it's not going to work. You could be very surprised. ---
**Author:** ![ndhcoder](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ndhcoder/32/63850_2.png) [@ndhcoder](https://discuss.elastic.co/u/ndhcoder)\ **Post date:** [March 6, 2020, 10:30am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/11 "2020-03-06T10:30:21Z")
Yep. I'll try that. Thanks you ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 3, 2020, 10:30am UTC](https://discuss.elastic.co/t/aggregations-large-data-in-real-time-in-elasticsearch-what-solution-is-the-best/222188/12 "2020-04-03T10:30:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.