Scan/Scroll performance and cache

Elastic Stack Elasticsearch
#1

We're running Elasticsearch 1.7.5 and we have an API that, given a few filters (API-level, an abstraction over what's stored in ES) returns a list of matching doc ids. This list can be 100k ids and that's why we're using a scan/scroll.

What I've noticed is that the first time we perform the scan/scroll with a few filters it takes ~25s to scroll over all the results (size=2k, 6 shards, query with 46k results). If we repeat the same query a second time it takes the same time, probably because we have replica=1 and it's hitting the other shard. The third time it takes 4s.

Why this difference? Ok, there's the filter cache, but that is also created with a normal search (not scan) that takes milliseconds. And I'm using _source=false, therefore it shouldn't be hitting the disk to retrieve data, because I only need ids.

Related question: is it ok to use scan/scroll for this? I only need ids, can I switch to a normal _search?

Yet another question: can I turn off things that bloat the response like the index name and the score that are in every hit?

Thank you

#2

At a guess, cause I don't know for sure, it's the FS cache keeping the files in out-of-heap RAM.

#3

Probably fs caching like mark says. You can use response filtering if you
want to remove bits of the response.

#4

Exactly, one reason why a search result set construction may be slow is relevance scoring. In ES 2.x you can use sort parameter _doc to switch off scoring while scrolling: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html

#5

Isn't _doc useful only for "scroll only" requests? My guess was that's not useful for scans, but i may be wrong (the doc is not clear about this IMHO)

#6

Oh, response filtering, thank you!

#7

That's possible. I'll monitor disk I/O then. Thank you

#8

Back to your response... FS caching of what exactly? Not of the indexes because a "normal" query runs in milliseconds, and that needs to read the whole reverse index too, at least of the two fields I'm stressing.

Thanks

#9

The OS caches often read files in free memory.

#10

Ok, but shouldn't a search take the same amount of time? I mean, a scan/scroll does 2 things:

  1. it filters the results, to find all possible matches, like a normal search
  2. it returns ids of every result, unlike a normal search

If the scan takes 2 order of magnitude more of a normal search, and point 1 is the same, it means point 2 is extremely slow. Since I use _source=false, does it mean it's reading the _source to get the _id?

Using fatrace I see there are a lot of reads on .fdt files (9k reads per shard, 6 shards, 13k results) that in my understanding doesn't store "field data" a-la ElasticSearch but the _source I guess. This would confirm my guess...

What about _uid? Can I ask that instead of _id? Would it be better?

Thank you in advance

#11

An update: using fincore I see that ES is reading .fdt files, even if I'm using _source=false in the scan/scroll. Isn't this a bug?

Thank you

#12