# Fetch 1 million records

**URL:** <https://discuss.elastic.co/t/fetch-1-million-records/220636>\
**Category:** Elasticsearch\
**Created:** [February 24, 2020, 11:23am UTC](https://discuss.elastic.co/t/fetch-1-million-records/220636 "2020-02-24T11:23:17Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![otherview](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otherview/32/38646_2.png) [@otherview](https://discuss.elastic.co/u/otherview)\
**Post date:** [February 24, 2020, 11:23am UTC](https://discuss.elastic.co/t/fetch-1-million-records/220636/1 "2020-02-24T11:23:17Z")

</div>

Hi !

I'm trying to fetch +1 Million records from elastic search.  
So far all approaches are still _slow_.

At the moment I'm using a sliced scroll search.

I just need a few fields from the documents, not the whole document. Is there a way I can optimize this ?

Thanks!

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [February 25, 2020, 4:15am UTC](https://discuss.elastic.co/t/fetch-1-million-records/220636/2 "2020-02-25T04:15:02Z")

</div>

I think [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-fields.html#modules-scripting-doc-vals) describes what you can try, but read the warnings about text fields.

[This looks like a similar question](https://discuss.elastic.co/t/do-doc-values-allow-for-sorted-scrolling-over-a-billion-of-documents/2596)

Let us know how it works out 🙂

---

<div class="post-metadata">

**Author:** ![Bernt\_Rostad](https://avatars.discourse-cdn.com/v4/letter/b/3ab097/32.png) [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)\
**Post date:** [February 25, 2020, 5:44am UTC](https://discuss.elastic.co/t/fetch-1-million-records/220636/3 "2020-02-25T05:44:53Z")

</div>

> [@otherview](#):
>
> I just need a few fields from the documents, not the whole document. Is there a way I can optimize this ?

You can use [source filtering](https://www.elastic.co/guide/en/elasticsearch/reference/current//search-request-body.html#request-body-search-source-filtering) to include or exclude fields from the result.

For instance, let's say I have an index my\_published\_stories and want to loop over all but just fetch the publishing date (pubdate) and processing time, then I'd do something like this:

```
GET my_published_stories/_search
{
  "_source": ["pubdate", "processing_time"]
}

```

Hope this answered your question.

---

<div class="post-metadata">

**Author:** ![otherview](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otherview/32/38646_2.png) [@otherview](https://discuss.elastic.co/u/otherview)\
**Post date:** [February 25, 2020, 3:47pm UTC](https://discuss.elastic.co/t/fetch-1-million-records/220636/4 "2020-02-25T15:47:55Z")

</div>

Hi @Bernt_Rostad

Yep, I'm already doing that 😄 Thanks!

---

<div class="post-metadata">

**Author:** ![otherview](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otherview/32/38646_2.png) [@otherview](https://discuss.elastic.co/u/otherview)\
**Post date:** [February 25, 2020, 3:49pm UTC](https://discuss.elastic.co/t/fetch-1-million-records/220636/5 "2020-02-25T15:49:43Z")

</div>

Thanks for that @rugenl !

I'm going to give the doc values a go! I'll post the results when I have some! 😄

---

<div class="post-metadata">

**Author:** ![otherview](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otherview/32/38646_2.png) [@otherview](https://discuss.elastic.co/u/otherview)\
**Post date:** [March 2, 2020, 11:32am UTC](https://discuss.elastic.co/t/fetch-1-million-records/220636/6 "2020-03-02T11:32:18Z")

</div>

hi @rugenl!

I've started using Sliced Scroll using doc\_values and no major improvements.. Any other ideas ?

Thanks!

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [March 2, 2020, 4:33pm UTC](https://discuss.elastic.co/t/fetch-1-million-records/220636/7 "2020-03-02T16:33:09Z")

</div>

Ok, what language are you using?

I put some timing in my scripts to measure what time I was waiting on the scroll vs. when I was processing the data. The waiting time includes network time, but it let me know whether to improve my search or my process.

Powershell was MUCH slower than the Python Elasticsearch DSL, like orders of magnitude slower.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 30, 2020, 4:48pm UTC](https://discuss.elastic.co/t/fetch-1-million-records/220636/8 "2020-03-30T16:48:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.