Alternative search using “Scroll” API and “Search After” API for real-time queries

Elastic Stack Elasticsearch
1

I need to send real-time queries for getting data from the dynamic index, that can be changed anytime. I speak about millions of docs in the index.

The usage of Scroll API was useful to me, given that it stored state and worked with consistency data. But on a big amount of records - it had a very bad performance. And depend on timeout parameter, that unacceptable for my purposes. Search After API looks good, but anyway it stateless, so it can lead to data loss or duplication of records in the sample.

Is there are some way to resolve this? Aggregating/filtering queries not acceptable in my situation.

2

Welcome!

Using the Scroll API is the way to go.

But on a big amount of records - it had a very bad performance.

What do you mean?

3

I'm using Scroll API for simulation of pagination in my back-end. So I don't know in advance when my user needs to get a new page with data. In this way, timeout of Scroll context need to be as max as possible, but I think it isn't the acceptable road of resolving my problem(for example if I set timeout = 1h)...

4

I see. Can't really think of an option you can have.
I'd probably give a 10 minutes timeout (or whatever) but anytime I'm detecting that the user is not going to the next page (like running another search, exiting the app, ...) I'd cancel its scroll id explicitly.

1 Like
5

Got it, thank you! I agree with your opinion.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.