Fetching large data from Elasticsearch in Node.js

Elastic Stack Elasticsearch
1

I am fetching about 60,000 documents from an index ABC in Elasticsearch (v7) through their Node.js client. I tried using their recommended Scroll API but it took almost 20s to do so. Next, I increased the max_window_size of the index ABC to 100,000 (from the default 10,000 ) and the query took 5s .

I am running Elasticsearch v7 on a 8-core, 32GB-RAM machine (1-node cluster). I am not performing any aggregations and my query is just a simple GET request to fetch all documents from the index ABC . Is there any way to speed this up to less than one second through their Node.js client?

My code in Express

const { body } = await client.search({
          index: 'ABC',
          size: 60000,
          body: {
            query: {
              match_all: {}
            }
          },
        });
        res.send(body.hits.hits);

If there's no other way to cut the time down, should I cache the entire response in memory and let the UI read from it rather than hitting ES directly (the data only updates once per day, I might not need to perform cache validations). Should I look into Redis for this?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.