How to check if ES has more documents when paginating with search after API?


#1

I'm trying to implement pagination of my GraphQL API with ElasticSearch's pagination feature. My pagination implementation in GraphQL API follows Relay specification. As far as I know, search after feature in ES only returns cursors in Relay specification, which can be used to tell ES which item you fetched in the last request.
In Relay specification, client could also check if there are more items in next page with hasNextPage property.
But how can I implement hasNextPage API without this information from ES?

Composite aggregation has similar pagination feature as search after API, and does not have properties like hasNextPage either.

Only method I can think of is including integer id in cursor. Pagination initially starts without cursor, so assign 1 to the first item. When the integer id becomes the same number as the total document document, there are no items left. This may not work properly if documents get added or removed in ES, though.

https://facebook.github.io/relay/graphql/connections.htm

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html#_after


(Murilo Godois de Castro Tavares) #2

I am in the same situation here.
For now the workaround I have in mind is to always query for page+1 elements, and returning only the first page'th elements to the user. If the query returns less than page+1 elements we know there's no next page. In other words, let's say the user asks for 10 elements. We ask ES for 11 elements. If ES returns 11 elements, then we return the first 10 to the user and set hasNextPage as true.

But this solution does not make me proud at all, and also has a small extra cost associated, so I wonder if there's any nicer way of doing this using ElasticSearch.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.