# Do Doc Values allow for sorted scrolling over a billion of documents? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 12, 2015, 6:46pm UTC](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596 "2015-06-12T18:46:03Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![lisak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisak/32/618_2.png) [@lisak](https://discuss.elastic.co/u/lisak)\ **Post date:** [June 12, 2015, 6:46pm UTC](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596/1 "2015-06-12T18:46:03Z")
Hi, I need to perform scrolling over a billion of docs, sorted by just one field `Foo`. If I format this single field `Foo` (+ possibly but I guess not necessarily all those fields retrieved by scroll) as "doc\_values", is it gonna work? Currently I can perform sorted scrolling over like 10-20 millions of documents, more of them cause OutOfMemory error. I currently have a billion of documents across 100 indices (500 shards) across 3 nodes. ---
**Author:** ![Harlin\_ES](https://avatars.discourse-cdn.com/v4/letter/h/977dab/32.png) [@Harlin\_ES](https://discuss.elastic.co/u/Harlin_ES)\ **Post date:** [June 12, 2015, 8:05pm UTC](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596/2 "2015-06-12T20:05:54Z")
using doc\_values is probably the only way this will work, since it doesn't use any memory. Probably the OOM you got was caused by pulling too many field values into the field data cache and blowing it up. Doc\_values should keep everything on disk so you don't have to worry. Just remember that after changing the mapping format for the Foo field you will have to re-index all of your data in order for the doc-values to get pulled out. I have personally used doc\_values for queries hitting 1bil+ documents and it worked fine. ---
**Author:** ![lisak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisak/32/618_2.png) [@lisak](https://discuss.elastic.co/u/lisak)\ **Post date:** [June 14, 2015, 9:19pm UTC](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596/3 "2015-06-14T21:19:30Z")
I just reindexed 300 mils of docs in 100 indices (500 shards total) on 3 nodes (c4.large instances) with doc\_values on 4 fields. And I'm sort-scrolling over those 300 mils documents with just those 4 fields loaded and it takes like 720 hours with CPU on 100% ... It seems to have enough memory on Heap but I suspect that remaining off-heap 1.8GB of OS memory might be the bottleneck... Also ```auto "query_total": 974748 "query_time_in_millis": 37625736 ``` Average of 38 milis per query seems to be a little slow on such amount of data... I'm also surprised that after I changed the critical fields to `doc_values` there is almost zero IO load... Any idea how sorted scrolling can be optimized? Would increasing field data cache help? ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 15, 2015, 12:52pm UTC](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596/4 "2015-06-15T12:52:13Z")
Could you please provide us with your elasticsearch version and the stack trace of the OOME? Also, how many documents do you retrieve per page? > [@lisak](#): > > I'm also surprised that after I changed the critical fields to doc\_values there is almost zero IO load... It might happen eg. if all your data fit into the filesystem cache. ---
**Author:** ![lisak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisak/32/618_2.png) [@lisak](https://discuss.elastic.co/u/lisak)\ **Post date:** [June 15, 2015, 2:16pm UTC](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596/5 "2015-06-15T14:16:34Z")
Hi, I cannot reproduce the OOME stacktrace since I reindexed everything to `doc_values` and now it is just slow, CPUs on 100% and no other obvious bottlneck like IO wait or GC chocking, probably more heap memory would help. We have 300 mils of documents in 100 indices on 3 nodes using 5 shards, each node is a c4.large instance, each ES has 2048MB ram. And default configuration, I'm not exactly sure whether field data cache would improve it radically. I'm using ES 1.5.1 However I tried to set scroll batch size to 2,10, 40 to 200 which should roughly correspond to 0.1 to 5MB after multiplying the number of shards the request might hit. Not more... And it is still insanely slow, I played with all day and sorted scroll over 300mills of documents with just 4 `doc_values` field loaded would take hundreds of hours. It was scanning like 100 - 400 documents per second, not more. I'm actually not sure if ec2 c4.large instance storage has any filesystem cache and how big is it. But it most probably does. It wouldn't make sense. We are using instance storages, not EBS. ---
**Author:** ![lisak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisak/32/618_2.png) [@lisak](https://discuss.elastic.co/u/lisak)\ **Post date:** [June 15, 2015, 4:31pm UTC](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596/6 "2015-06-15T16:31:21Z")
Btw I started to call `optimize` before scrolling and I optimized `scroll search size` (increased) and now it seems to be scrolling 1000 small docs per second which might be sufficient ! Thanks ---
**Author:** ![lisak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisak/32/618_2.png) [@lisak](https://discuss.elastic.co/u/lisak)\ **Post date:** [June 17, 2015, 7:17pm UTC](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596/7 "2015-06-17T19:17:58Z")
I've asked another question about [internal workings of sorted scroll](https://discuss.elastic.co/t/sorted-scroll-internal-workings-for-performance-optimizations/23866). It doesn't really perform well and I can't improve it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:06am UTC](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596/8 "2017-07-06T00:06:57Z")