I have a index with around 50Million data points, where I have a ID for each document. I need unique id and its count , as it would be more than 10000, I used scrolling but unexpecteded, the scrolling gives me the same scrollid in the iteration > data = es.search(index="ttd-conversions-2019-05*", …

Don't use scroll. Use a regular search using the composite agg and then another search using a composite agg with the after parameter returned in the previous result. Repeat as necessary

Scroll in ElasticSearch Aggregation

Elastic Stack Elasticsearch

Vinothkumar_Ganeshan (Vinothkumar Ganeshan) November 29, 2019, 3:53pm 7

Thanks for your time and effort. I would be helpful, if you could share your insights for the following problem Count in other index based on Current Index field

Topic		Replies	Views
Response 400 while using Scroll API with Python - scroll_id too long Elasticsearch	1	1445	August 14, 2019
Scroll id is not changing while querying Elasticsearch	2	4898	December 8, 2017
Do unique/reusable _scroll_ids exist? Elasticsearch	4	1509	July 6, 2017
Retrieving millions of large documents Elasticsearch	7	370	September 25, 2023
Same scroll id Elasticsearch	3	1783	July 5, 2017

Scroll in ElasticSearch Aggregation

Related topics