Scroll documents with ElasticSearch 8.9 for dotnet

Elastic Stack Elasticsearch
1

Hi,

I am looking for some assisstance in getting scrolling working for ES client (Version 8.9) for .NET. I have tried a few things but cannot get them to work, I have tried as per example in documentation for v7.17 but it wouldn't work.

Whilst trying to fetch 56760 documents the main issues are:

  • ScrollResponse returns ScrollId as null despite not being done with the results. This means I cannot use the ScrollId from the response to continue with the data.
  • When using the same ScrollId from initial Search I encounter the error: (404: No search context found for id )

Because of this I can get only 20000 of the 50000+ documents. My current code is as follows:

List<TDocument> hits = new List<TDocument>();
var result = _client.Search<TDocument>(s => s
        .Index(indexName)
        .Size(10000)
        .Query(q => q
            .Range(r => r
                .NumberRange(nr => nr
                    .Field(f => f.tstamp)
                    .Gte(startTicks)
                    .Lt(endTicks))))
        .Scroll(new TimeSpan(0, 30, 0)));

hits.AddRange(extractHitsFromResult(result));
if (result.Hits.Count == 10000)
{
    var scrollRequest = new ScrollRequest() { ScrollId = result.ScrollId };
    var scrollHits = 0;
    do
    {
        var scrollResult = _client.Scroll<TDocument>(scrollRequest);
        hits.AddRange(extractHitsFromResult(scrollResult));
        scrollHits = scrollResult.Hits.Count;
    }
    while (scrollHits > 0);
}

But I also saw failure when implementing like this:

List<TDocument> hits = new List<TDocument>();
var result = _client.Search<TDocument>(s => s
        .Index(indexName)
        .Size(10000)
        .Query(q => q
            .Range(r => r
                .NumberRange(nr => nr
                    .Field(f => f.tstamp)
                    .Gte(startTicks)
                    .Lt(endTicks))))
        .Scroll(new TimeSpan(0, 30, 0)));

hits.AddRange(extractHitsFromResult(result));
if (result.Hits.Count == 10000)
{
    var scrollRequest = 
    var scrollHits = 0;
    var scrollId = result.ScrollId;
    do
    {
        var scrollResult = _client.Scroll<TDocument>(new ScrollRequest() { ScrollId = scrollId });
        hits.AddRange(extractHitsFromResult(scrollResult));
        scrollHits = scrollResult.Hits.Count;
        scrollId = scrollResult.ScrollId;
    }
    while (scrollHits > 0);
}

Any solutions or advice would be appreciated. I have collated my information from the documentation found here: ElasticSearch Searching Documents v8.9 and here ElasticSearch Scrolling Documents v7.17

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.