# Scan/Scroll performance and cache **URL:** \ **Category:** Elasticsearch\ **Created:** [March 11, 2016, 3:29pm UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173 "2016-03-11T15:29:39Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![vad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vad/32/3901_2.png) [@vad](https://discuss.elastic.co/u/vad)\ **Post date:** [March 11, 2016, 3:29pm UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/1 "2016-03-11T15:29:39Z")
We're running Elasticsearch 1.7.5 and we have an API that, given a few filters (API-level, an abstraction over what's stored in ES) returns a list of matching doc ids. This list can be 100k ids and that's why we're using a scan/scroll. What I've noticed is that the first time we perform the scan/scroll with a few filters it takes ~25s to scroll over all the results (size=2k, 6 shards, query with 46k results). If we repeat the same query a second time it takes the same time, probably because we have replica=1 and it's hitting the other shard. The third time it takes 4s. Why this difference? Ok, there's the filter cache, but that is also created with a normal search (not scan) that takes milliseconds. And I'm using \_source=false, therefore it shouldn't be hitting the disk to retrieve data, because I only need ids. Related question: is it ok to use scan/scroll for this? I only need ids, can I switch to a normal \_search? Yet another question: can I turn off things that bloat the response like the index name and the score that are in every hit? Thank you ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 12, 2016, 8:44am UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/2 "2016-03-12T08:44:36Z")
At a guess, cause I don't know for sure, it's the FS cache keeping the files in out-of-heap RAM. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [March 12, 2016, 2:07pm UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/3 "2016-03-12T14:07:27Z")
Probably fs caching like mark says. You can use response filtering if you want to remove bits of the response. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 12, 2016, 2:50pm UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/4 "2016-03-12T14:50:15Z")
> [@vad](#): > > Yet another question: can I turn off things that bloat the response like the index name and the score that are in every hit? Exactly, one reason why a search result set construction may be slow is relevance scoring. In ES 2.x you can use sort parameter `_doc` to switch off scoring while scrolling: [Request body search | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html) ---
**Author:** ![vad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vad/32/3901_2.png) [@vad](https://discuss.elastic.co/u/vad)\ **Post date:** [March 14, 2016, 10:35am UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/5 "2016-03-14T10:35:50Z")
Isn't `_doc` useful only for "scroll only" requests? My guess was that's not useful for scans, but i may be wrong (the doc is not clear about this IMHO) ---
**Author:** ![vad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vad/32/3901_2.png) [@vad](https://discuss.elastic.co/u/vad)\ **Post date:** [March 14, 2016, 10:36am UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/6 "2016-03-14T10:36:29Z")
Oh, response filtering, thank you! ---
**Author:** ![vad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vad/32/3901_2.png) [@vad](https://discuss.elastic.co/u/vad)\ **Post date:** [March 14, 2016, 10:38am UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/7 "2016-03-14T10:38:24Z")
That's possible. I'll monitor disk I/O then. Thank you ---
**Author:** ![vad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vad/32/3901_2.png) [@vad](https://discuss.elastic.co/u/vad)\ **Post date:** [March 15, 2016, 10:24am UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/8 "2016-03-15T10:24:45Z")
Back to your response... FS caching of what exactly? Not of the indexes because a "normal" query runs in milliseconds, and that needs to read the whole reverse index too, at least of the two fields I'm stressing. Thanks ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 15, 2016, 8:04pm UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/9 "2016-03-15T20:04:01Z")
The OS caches often read files in free memory. ---
**Author:** ![vad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vad/32/3901_2.png) [@vad](https://discuss.elastic.co/u/vad)\ **Post date:** [March 15, 2016, 8:17pm UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/10 "2016-03-15T20:17:34Z")
Ok, but shouldn't a search take the same amount of time? I mean, a scan/scroll does 2 things: 1. it filters the results, to find all possible matches, like a normal search 2. it returns ids of every result, unlike a normal search If the scan takes 2 order of magnitude more of a normal search, and point 1 is the same, it means point 2 is extremely slow. Since I use `_source=false`, does it mean it's reading the `_source` to get the `_id`? Using `fatrace` I see there are a lot of reads on `.fdt` files (9k reads per shard, 6 shards, 13k results) that in my understanding doesn't store "field data" a-la ElasticSearch but the `_source` I guess. This would confirm my guess... What about `_uid`? Can I ask that instead of `_id`? Would it be better? Thank you in advance ---
**Author:** ![vad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vad/32/3901_2.png) [@vad](https://discuss.elastic.co/u/vad)\ **Post date:** [April 4, 2016, 12:35pm UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/11 "2016-04-04T12:35:07Z")
An update: using [fincore](http://code.google.com/p/linux-ftools/) I see that ES is reading `.fdt` files, even if I'm using \_source=false in the scan/scroll. Isn't this a bug? Thank you ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:02pm UTC](https://discuss.elastic.co/t/scan-scroll-performance-and-cache/44173/12 "2017-07-05T23:02:35Z")