Getting inconsistent records while using From + Size pagination approach

Elastic Stack Elasticsearch
1

Hi,

We are using From + Size approach (Ref.) for pagination without any explicit sort field.
While using it, we noticed discrepancies in the records returned by the query. Sometimes, the records are getting overlapped and sometimes getting missed. For e.g. getting Record-1 for input: From = 0, Size= 5 (Page no. 1) and again getting Record-1 for input: From = 5, Size= 5 (Page no. 2)

Reason: Elastic search by default sort the records on basis of _score field which can be same for multiple records resulting in random ordering of records. Observed that _score is same for multiple records.

Also, tried using _doc as a sort field but got same behaviour. _doc is preferred for sorting as per this doc, _doc has no real use-case besides being the most efficient sort order. So if you don’t care about the order in which documents are returned, then you should sort by _doc..

I am now using some other field(custom field which is unique for our records) as a sorting field and getting correct records and pagination behaviour.

Below are my queries:

  1. Can you please suggest if there is any other way to tackle this issue? Also, what is the impact of using custom sort field on elastic search cluster?

  2. Can above mentioned issues occur with scroll based pagination approach if we have same score for more than 1000+ records?

Also, I think we should update the official elastic search doc for From + Size mentioning a note that _score field can be same and that can cause inconsistent behaviour in pagination.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.