# Get all 3 million ids of a type very quickly

**URL:** <https://discuss.elastic.co/t/get-all-3-million-ids-of-a-type-very-quickly/107013>\
**Category:** Elasticsearch\
**Created:** [November 9, 2017, 10:05am UTC](https://discuss.elastic.co/t/get-all-3-million-ids-of-a-type-very-quickly/107013 "2017-11-09T10:05:21Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![spotlight2001](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spotlight2001/32/54078_2.png) [@spotlight2001](https://discuss.elastic.co/u/spotlight2001)\
**Post date:** [November 9, 2017, 10:05am UTC](https://discuss.elastic.co/t/get-all-3-million-ids-of-a-type-very-quickly/107013/1 "2017-11-09T10:05:21Z")

</div>

i want to get all ids of a type from elasticsearch via http  
i have approx. 3 million ids

i am using scroll api  
scroll=1m  
size=10000  
{  
"query": {  
"match\_all": {  
}  
},  
"\_source": false,  
"sort": ["\_doc"]  
}

its fast in the beginning but gets very slow (4 minutes and more per scroll) later on. even with a higher scroll=5m time

increasing size:  
size=100000

- changing result window  
doesnt change it. its fast at the start and gets slow later on

what can i do?

why do i want to get all ids:  
i have a RDBMS with columns for lastmodified  
some import deletes records in my RDMBS  
i periodically query my RDBMS and update elasticsearch  
for deleting i get all ids from RBDMS  
all ids from elasticsearch  
and delete from elasticsearch all documents by id which are not (or no longer) in RDBMS

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 9, 2017, 10:14am UTC](https://discuss.elastic.co/t/get-all-3-million-ids-of-a-type-very-quickly/107013/2 "2017-11-09T10:14:04Z")

</div>

Why not keep track of deleted documents in the RDBMS, e.g. through a trigger, so that you can manage deletes based on this instead of having to compare all IDs, which will scale badly?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 7, 2017, 10:14am UTC](https://discuss.elastic.co/t/get-all-3-million-ids-of-a-type-very-quickly/107013/3 "2017-12-07T10:14:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.