How to use Scroll in Rails/Ruby to get more than 10,000 records?

Elastic Stack Elasticsearch
1

I need to search in my 25,000 records. But elasticsearch allow only 10,000 (from and size is the same). How can I make search and get results more than 10,000?
There is scroll api, but I don't understand how to use it.
EXAMPLE: result = Product.search('red',{scroll: '5m'}) .records
How to iterate after it?

2

Have a look at https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html

You can optionally modify index.max_result_window index settings to 25000 if you really need it.
But this comes with a memory pressure price. See https://www.elastic.co/guide/en/elasticsearch/reference/6.7/index-modules.html

As the documentation says, you can use:

About your question about scroll, the documentation I just linked to describes how it works. Is there anything unclear in it?

1 Like
3

In Rails (elasticsearch-rails):

products = []
after = 0
loop do
  query = {
    size: query_size,
    query: {
      multi_match: {
        query: "black",
        fields: [ "description", "title", "information", "params" ]
      }
    },
    search_after: [after],
    sort: [ {id: "asc"} ]
  }
  results =  Product.search(query).records
  results_amount = results.size
  products += results
  after += 10000

  break if results_amount == 0
end
5

Thank you! search_after documentation is what I was needed.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.