# How to retrieve millions of records very effectively at a time and display them in grid using NEST client with elastic search?

**URL:** <https://discuss.elastic.co/t/how-to-retrieve-millions-of-records-very-effectively-at-a-time-and-display-them-in-grid-using-nest-client-with-elastic-search/161447>\
**Category:** Elasticsearch\
**Created:** [December 19, 2018, 6:04am UTC](https://discuss.elastic.co/t/how-to-retrieve-millions-of-records-very-effectively-at-a-time-and-display-them-in-grid-using-nest-client-with-elastic-search/161447 "2018-12-19T06:04:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![karan.j.shah](https://avatars.discourse-cdn.com/v4/letter/k/9de053/32.png) [@karan.j.shah](https://discuss.elastic.co/u/karan.j.shah)\
**Post date:** [December 19, 2018, 6:04am UTC](https://discuss.elastic.co/t/how-to-retrieve-millions-of-records-very-effectively-at-a-time-and-display-them-in-grid-using-nest-client-with-elastic-search/161447/1 "2018-12-19T06:04:46Z")

</div>

Hello Everyone,  
Currently, I am using ScrollAll Method Of NEST Client to retrieve millions of records from elastic search. Below I have mentioned Code for this:

```
                     var numberOfShards=3;
                     objElasticClient.ScrollAll<PubmedIndexFields>("1m", numberOfShards, s => s
                    .MaxDegreeOfParallelism(numberOfShards / 2)
                    .Search(search => search
                        .Query(
                        q => q.QueryString(c => c
                        .Boost(1.1)
                        .Query(query.ToString())
                        .Analyzer(ElasticIndexConfiguration.CustomAnalyzerName)
                        .AllowLeadingWildcard()
                         )).Size(PageSize)
                        )
                ).Wait(TimeSpan.FromMinutes(5), r =>
                {

                     var lstrecords = r.SearchResponse.Documents.ToList<PubmedIndexFields>();
                    TaskPool.AddRange(lstrecords.Select((t, i) =>
                    Task<List<IndexEntityFields>>.Factory.StartNew((obj) =>
                    {
                        tokenSource.Token.ThrowIfCancellationRequested();
                        var loc = (int)obj;
                        var lstobject = lstrecords.ElementAt(loc);
                        var indexEntityFieldses = new List<IndexEntityFields>();
                        var op = new IndexEntityOperation();
                        op.Parse_ElasticSearch(lstobject);
                        indexEntityFieldses.Add(op.Fields);
                        return indexEntityFieldses;
                    }, i, tokenSource.Token)));

                    Interlocked.Add(ref seenDocuments, r.SearchResponse.Hits.Count);
                     }

```

But Sometimes I did not get the proper result set.

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [December 28, 2018, 1:07am UTC](https://discuss.elastic.co/t/how-to-retrieve-millions-of-records-very-effectively-at-a-time-and-display-them-in-grid-using-nest-client-with-elastic-search/161447/2 "2018-12-28T01:07:31Z")

</div>

Using the Scroll API, and the ScrollAll observable helper within NEST is an efficient method to retrieve a large volume of documents.

I would question the value of displaying millions of records to an end user in a UI; sounds like a good use case to allow the end user to search those records to filter to those most relevant?

> [@karan.j.shah](#):
>
> ```auto
> TaskPool.AddRange(lstrecords.Select((t, i) =>
> Task<List<IndexEntityFields>>.Factory.StartNew((obj) =>
> 
> ```

There are a couple of questions about this code:

1. I'm not sure why a `Task<T>` is kicked off to perform a conversion of the document result; there is nothing _awaiting_ the result of these asynchronous operations in the example provided. Looks like a synchronous operation would suffice here as I would imagine this conversion to be very quick.

2. The LINQ `.Select(...)` and `ElementAt(...)` could be replaced with a simple `foreach(...)` over `lstrecords`, or a `for` loop, since the collection is `List<T>`.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 25, 2019, 1:07am UTC](https://discuss.elastic.co/t/how-to-retrieve-millions-of-records-very-effectively-at-a-time-and-display-them-in-grid-using-nest-client-with-elastic-search/161447/3 "2019-01-25T01:07:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.