# Use search or scroll for large dataset which needs aggregations

**URL:** <https://discuss.elastic.co/t/use-search-or-scroll-for-large-dataset-which-needs-aggregations/346578>\
**Category:** Elasticsearch\
**Created:** [November 6, 2023, 10:45pm UTC](https://discuss.elastic.co/t/use-search-or-scroll-for-large-dataset-which-needs-aggregations/346578 "2023-11-06T22:45:38Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nboisnea1](https://avatars.discourse-cdn.com/v4/letter/n/2bfe46/32.png) [@nboisnea1](https://discuss.elastic.co/u/nboisnea1)\
**Post date:** [November 6, 2023, 10:45pm UTC](https://discuss.elastic.co/t/use-search-or-scroll-for-large-dataset-which-needs-aggregations/346578/1 "2023-11-06T22:45:38Z")

</div>

Hi! I'm new to Elasticsearch and I have a particular use case for which I don't know if I should use a basic search or a scroll search.  
I have an index in which I periodically save a copy of JSON documents. Each JSON document features an ID that refers to an external data entity. Documents are inserted every day, but not for every entity's ID systematically.  
At a certain date A, I want to retrieve and process the most recent entry for each existing entity ID. In other words, I want to filter documents for which the date is before date A, sort them by descending date, and group by ID for which I want to keep only the first document.

Here are the two solutions:

- Using search with a `collapse` clause on the ID, then iterating over pages until I have fetched all the documents corresponding to each ID.
- Using scroll to fetch all documents whose date is less than date A sorted by descending date, then manually handling the grouping because I can't use `collapse` when using scroll.

The first solution does not seem suitable since I could deal with very large datasets for which the scroll API seems to be made for.  
But scrolling does not really seem convenient given I can have hundreds of versions for each ID, dating years before date A, when I only need the most recent version.

What would be the best approach for this use case?  
Thanks for any hints, maybe I'm not headed in the right direction here!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 4, 2023, 10:46pm UTC](https://discuss.elastic.co/t/use-search-or-scroll-for-large-dataset-which-needs-aggregations/346578/2 "2023-12-04T22:46:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.