# Scroll in ElasticSearch Aggregation **URL:** \ **Category:** Elasticsearch\ **Created:** [November 28, 2019, 4:12pm UTC](https://discuss.elastic.co/t/scroll-in-elasticsearch-aggregation/209885 "2019-11-28T16:12:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Vinothkumar\_Ganeshan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinothkumar_ganeshan/32/46534_2.png) [@Vinothkumar\_Ganeshan](https://discuss.elastic.co/u/Vinothkumar_Ganeshan)\ **Post date:** [November 28, 2019, 4:12pm UTC](https://discuss.elastic.co/t/scroll-in-elasticsearch-aggregation/209885/1 "2019-11-28T16:12:17Z")
I have a index with around 50Million data points, where I have a ID for each document. I need unique id and its count , as it would be more than 10000, I used scrolling but unexpecteded, the scrolling gives me the same scrollid in the iteration ``` > data = es.search(index="ttd-conversions-2019-05*", scroll='1m', body= bodys) > sid = data['_scroll_id'] > scroll_size = len(data['hits']['hits']) > count = list() > tdid = list() > while(scroll_size > 0): > log_time = list() > tdid = list() > print('Scroll id', sid) > page = es.scroll(scroll_id= sid, scroll = '1m') > sid = page['_scroll_id'] > > count = list() > tdid = list() > for i in data['aggregations']['2']['buckets']: > count.append(i['doc_count']) > tdid.append(i['key']) > > scroll_size = len(page['hits']['hits']) > with open(save_path + "/out.csv", "a", newline="") as f: > writer = csv.writer(f) > writer.writerows(zip(count, tdid)) ``` Please let me know, this same code works fine for search with scroll, but aggregation repeats the same scroll id. Thanks for your help. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 28, 2019, 4:49pm UTC](https://discuss.elastic.co/t/scroll-in-elasticsearch-aggregation/209885/2 "2019-11-28T16:49:16Z")
The scroll api is used to scroll through documents, not aggregations. To look at options for large numbers of unique terms try run this [wizard](https://plnkr.co/edit/4u5CscRfiUOQwUDSDRgC?p=preview) to pick the right approach ---
**Author:** ![Vinothkumar\_Ganeshan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinothkumar_ganeshan/32/46534_2.png) [@Vinothkumar\_Ganeshan](https://discuss.elastic.co/u/Vinothkumar_Ganeshan)\ **Post date:** [November 28, 2019, 4:51pm UTC](https://discuss.elastic.co/t/scroll-in-elasticsearch-aggregation/209885/3 "2019-11-28T16:51:43Z")
Hi, Thanks for your respone. I understand that scroll id is to scroll through the documents, so i'm expecting that in scroll\_id would change for the next iterations, but it doesn't and always gives the first 10 results. I'm confused in it. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 28, 2019, 5:13pm UTC](https://discuss.elastic.co/t/scroll-in-elasticsearch-aggregation/209885/4 "2019-11-28T17:13:16Z")
Aggregations summarize the entire result set, not the current page of documents. If you want to page through aggregation results you need to see my previous advice. ---
**Author:** ![Vinothkumar\_Ganeshan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinothkumar_ganeshan/32/46534_2.png) [@Vinothkumar\_Ganeshan](https://discuss.elastic.co/u/Vinothkumar_Ganeshan)\ **Post date:** [November 29, 2019, 10:23am UTC](https://discuss.elastic.co/t/scroll-in-elasticsearch-aggregation/209885/5 "2019-11-29T10:23:10Z")
Thank you, Based on Wizard, which suggested the Composite Aggregation ``` GET ttd-conversions-2019-05*/_search?scroll=1m { "aggs" : { "my_buckets": { "composite" : { "sources" : [ { "TDID": { "terms" : { "field": "TDID"} }}]}}} } ``` Well, it returns the result with TDID and Count only for the first scroll, I request based on scroll\_id on next. It doesn't return the TDID with Count but just lists the same documents. I'm bit confused. Sorry if I misunderstood something. Thanks for your support, I means a lot ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [November 29, 2019, 10:25am UTC](https://discuss.elastic.co/t/scroll-in-elasticsearch-aggregation/209885/6 "2019-11-29T10:25:09Z")
Don't use scroll. Use a regular search using the composite agg and then another search using a composite agg with the `after` parameter returned in the previous result. Repeat as necessary ---
**Author:** ![Vinothkumar\_Ganeshan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinothkumar_ganeshan/32/46534_2.png) [@Vinothkumar\_Ganeshan](https://discuss.elastic.co/u/Vinothkumar_Ganeshan)\ **Post date:** [November 29, 2019, 3:53pm UTC](https://discuss.elastic.co/t/scroll-in-elasticsearch-aggregation/209885/7 "2019-11-29T15:53:24Z")
Thanks for your time and effort. I would be helpful, if you could share your insights for the following problem [Count in other index based on Current Index field](https://discuss.elastic.co/t/count-in-other-index-based-on-current-index-field/209971) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 27, 2019, 3:53pm UTC](https://discuss.elastic.co/t/scroll-in-elasticsearch-aggregation/209885/8 "2019-12-27T15:53:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.