# Slow results retrieval

**URL:** <https://discuss.elastic.co/t/slow-results-retrieval/157178>\
**Category:** Elasticsearch\
**Created:** [November 17, 2018, 1:24pm UTC](https://discuss.elastic.co/t/slow-results-retrieval/157178 "2018-11-17T13:24:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![TDZ](https://avatars.discourse-cdn.com/v4/letter/t/ccd318/32.png) [@TDZ](https://discuss.elastic.co/u/TDZ)\
**Post date:** [November 17, 2018, 1:24pm UTC](https://discuss.elastic.co/t/slow-results-retrieval/157178/1 "2018-11-17T13:24:52Z")

</div>

So I have 5 billion docs on 5 shards in 1 index, all on one machine, each shard has 1 segment.  
ES is running on 4 cpus with 26GB ram and 18GB of heap for ES.  
Each doc has 4 ints and 2 floats.  
Im running a query that uses range query on one field and a terms query on another, both are in the filter part of a bool query.  
Then I try to retrieve 500k results using the scroll API with 5 slices (I have 5 threads running at the same time, one per slice), Im fetching only one int field using \_source for each result, im using a 9K page size and im using the transport client for java.  
Its taking me ~50 seconds.... Does that make sense or am I doing something wrong?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 18, 2018, 11:37am UTC](https://discuss.elastic.co/t/slow-results-retrieval/157178/2 "2018-11-18T11:37:50Z")

</div>

> [@TDZ](#):
>
> with 26GB ram and 18GB of heap for ES

It is generally recommended to give no more than 50% of available RAM to heap. Elasticsearch requires off-heap memory for optimal performance.

> [@TDZ](#):
>
> Im fetching only one int field using \_source for each result

What if you instead fetch the full document so Elasticsearch do not need to parse it?

---

<div class="post-metadata">

**Author:** ![TDZ](https://avatars.discourse-cdn.com/v4/letter/t/ccd318/32.png) [@TDZ](https://discuss.elastic.co/u/TDZ)\
**Post date:** [November 19, 2018, 8:13pm UTC](https://discuss.elastic.co/t/slow-results-retrieval/157178/3 "2018-11-19T20:13:13Z")

</div>

Done both, didnt change anything

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 19, 2018, 8:49pm UTC](https://discuss.elastic.co/t/slow-results-retrieval/157178/4 "2018-11-19T20:49:18Z")

</div>

What does disk I/O and iowait look like during retrieval?

---

<div class="post-metadata">

**Author:** ![TDZ](https://avatars.discourse-cdn.com/v4/letter/t/ccd318/32.png) [@TDZ](https://discuss.elastic.co/u/TDZ)\
**Post date:** [November 19, 2018, 9:21pm UTC](https://discuss.elastic.co/t/slow-results-retrieval/157178/5 "2018-11-19T21:21:24Z")

</div>

Dont know about iowait but I assume its not a problem since im on an ssd, io is 300 iops and 38~MBs in spikes, thats the highest spike.  
But I think thats besides the point, I want to use the above mentioned search for real time usages, so I wonder whats a reasonable expectation? For example, can ES provide 100k results in 2-3 seocnds? What about 500k?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 17, 2018, 9:21pm UTC](https://discuss.elastic.co/t/slow-results-retrieval/157178/6 "2018-12-17T21:21:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.