Scrolling API Seems Inconsistent

Elastic Stack Elasticsearch
1

I'm using the .NET NEST client in my Azure function to retrieve data from our Elastic account. Some of the SearchAsync requests I make exceed the 10,000 search result limit and as such I have to use the ScrollAsync request after it to get the rest. I've noticed that sometimes the ScrollAsync request doesn't return the results that it should be. The request is successful with a 200 code but just returns 0 documents instead of the 2,000 more that it should be. I'd say this happens maybe like 20% of the time.

Has anyone else experienced this? Any recommendations or suggestions? I'm happy to provide code snippets if it'd be helpful. Thanks.

2

Yeah, this kind of issue with ScrollAsync in Elasticsearch does happen, and it can be tricky to catch. A few things that might be causing it:

Scroll expiration – If there’s too much time between scroll requests (like more than 1 minute), Elasticsearch drops the scroll context and just returns nothing.

Reusing the scroll ID – If you’re using the same ElasticClient for multiple scrolls or doing them in parallel, that can break the scroll session.

Delays in Azure Functions – Cold starts or unexpected delays can let the scroll expire before the next request is made.

Other client operations in between – If the client is used for something else between scroll calls, that can also interfere and clear the scroll context.

3

Hi @Moose,

I don't think this is a issue with the Elasticsearch .NET client. An expired scroll timeout would be my guess as well (hard to tell without server side logs).

Reusing the same ElasticClient instance across scrolls or multiple scrolls in parallel can invalidate the scroll.

It is highly recommended to reuse a single instance of the client! The Elasticsearch .NET client is stateless. Parallel requests using the same instance is absolutely no problem at all.

If other operations are executed on the same client between scrolls, it can clear the scroll context.

This is not correct either.

4

Hi @Rafa_Silva and @flobernd thank you for the replies.

I do not think the scroll context is expiring. I'm using .Scroll(10) in my SearchAsync request so it should be staying for 10 minutes. I am also using ClearScrollAsync after my SearchAsync request or after my ScrollAsync request (if scrolling was needed) to ensure that the context isn't kept open needlessly.

In regards to your second point, I am using the same ElasticClient for all requests so there may be something going on there. @flobernd says the opposite though so I'm not certain. I can give it a try and see what happens at the least.

Regarding the point about Azure functions. I probably should have specified that I'm experiencing this while still doing local testing in Visual Studio. The end goal of this program is an Azure function but I haven't gotten that far yet, sorry for misguiding.

Fortunately I am doing that. I have an if conditional between the SearchAsync and ScrollAsync requests that determines if the latter is required, so there should be no meaningful delay between them. The entire SearchAsync and follow up ScrollAsync requests takes like 30 seconds right now.

5

Since I'm the current maintainer of the .NET client, I can definitely guarantee you that there is no context preserved on client-side and that using different client instances for each request will do nothing besides lowering your performance and increasing your resource consumption :sweat_smile:

The value specifies the amount of milliseconds, not minutes :sweat_smile:

You can use .Scroll(Timespan.FromMinutes(10)) instead to get the desired result.

6

Oh wow, that could certainly be the problem. Let me try changing it to actual minutes then haha. I've found the documentation for this NEST client very difficult to find so a lot of this I've had to go off of on my own. So I didn't realize that was for milliseconds. The intellisense in Visual Studio doesn't specify, it just says "Time".

7

Good observation, since it is in milliseconds it may indeed be causing the problem. Test it and let us know the result, I hope everything works for you

8

I'll be keeping an eye on it during my testing over the next week or so and will report back on my results.

1 Like
9

@flobernd @Rafa_Silva I haven't had a ton of time to test out my app as much as I wanted to since my last message on here, but so far so good. I do think the mistaken timeout was probably my issue. I'll post again later when I've had more time to confirm this.

10

Thanks for reporting back here!

11

Thanks for the report, I hope it really solved your problem