I have a index with around 50Million data points, where I have a ID for each document. I need unique id and its count , as it would be more than 10000, I used scrolling but unexpecteded, the scrolling gives me the same scrollid in the iteration > data = es.search(index="ttd-conversions-2019-05*", …

Don't use scroll. Use a regular search using the composite agg and then another search using a composite agg with the after parameter returned in the previous result. Repeat as necessary

Scroll in ElasticSearch Aggregation

Elastic Stack Elasticsearch

Vinothkumar_Ganeshan (Vinothkumar Ganeshan) November 29, 2019, 3:53pm 7

Thanks for your time and effort. I would be helpful, if you could share your insights for the following problem Count in other index based on Current Index field

Topic		Replies	Views
Response 400 while using Scroll API with Python - scroll_id too long Elasticsearch	1	1449	August 14, 2019
Scroll id is not changing while querying Elasticsearch	2	4918	December 8, 2017
Do unique/reusable _scroll_ids exist? Elasticsearch	4	1511	July 6, 2017
Retrieving millions of large documents Elasticsearch	7	381	September 25, 2023
Same scroll id Elasticsearch	3	1787	July 5, 2017

Scroll in ElasticSearch Aggregation

Related topics