Storage limit and retrieval size

Elastic Stack Elasticsearch
1

Hi

  • I have a scenario where need to store from 10 to more than 100000 records in a single index and use
    like a cache provider too.
  • In search I need to retrieve from 20 to 10000 records.
  • What will be the best way to implement these scenario with dynamically changing storage limit and
    size for retrieval / any other way without dynamic change?
    • Without causing any memory overhead

Currently I am setting

  1. "index.max_result_window" : 100000
  2. Size : 100000

  • This may lead to memory issues as per earlier discussions

  • Scroll API how it will work?

Thanks
Aneesh

2

I need to retrieve from 20 to 10000 records.

I'm curious about this. Why do you need that? What is the process after you get 100000 records? What the user is going to see?

If you really need to do that and that the size is limited to let say 100000, then I'd probably run 2 queries.

The first one with size=0 just to count the actual number of results.
Then size=what you got from the first call to get all records. If more than xxx (that you have to determine depending on your hardware), use scroll to get more records than xxx. (Sliced scroll might help as well).

My 2 cents

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.