Fetching results beyond 10k - scroll/searchAfter

Elastic Stack Elasticsearch
1

We are on elastic 7.8, we have use case to fetch beyond 10k results are in interactive mode from users on pretty decent scale (say ~40 concurrent hits)

We have stumbled upon

  1. scroll (everytime open a scroll, retrieve and close the scroll)
  2. searchAfter (sorted by _id )

We did Load Testing with above both options
Config:

  1. Data Nodes -6 (4cores, 64G RAM, 1TB)
  2. Co-ordinator - 2 (4cores, 32G RAM)
  3. 60 Concurrent Threads doing search on cluster retrieving top 5000 results
  4. Index - 20M docs, 30 shards 2 replicas

Observations:

  1. Search is really bad with latencies over 40 secs
  2. With scrolls/searchAfter even regular searches are getting slower
  3. If disable scrolls/searchAfter, regular searches are getting better ~3-4 secs under load
  4. We dont see CPU/RAM being spiked up

Questions:

  1. Can someone throw light how scrolls/searchAfter are so expensive
  2. As our resources are under utilized, can someone point us to settings to be changes from default which are otherwise defaulted to conservative numbers?
2

Elasticsearch is not always limited by CPU, and can often instead be limited by disk I/O. What kind of storage do you have? Have you monitored disk I/o and iowait?

3

dont see much spikes in IO read/Writes as well, we are on ssd 1TB local storage per node

4

How large is the index (in GB)? What is the average shard size?

5

~500G for entire index and ~8G per shard

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.