How fast can elastic search records be read?

Elastic Stack Elasticsearch
1

We have a system that from one side writes information, and on the other sides reads the data.
I synchronize the data based on a timestamp. I read data from a point in time, arrange it according to the timestamp and continue reading from the last timestamp I got.
I noticed that if I always read the latest data, i miss some data which is written out of order, so instead I added a delay of 1 minute, and now everything is fine.
My question is, what is the minimal delay I should use? In other words, what is the "settling time" of written records, and does it depend on the throughput of data?

With Best Regards,
Arik Halperin

2

Once you have the response of elasticsearch, it can take up to 1s before the document is actually searchable. This is called refresh.
You can also make your index query wait for the next refresh to happen in order to send back the index response.

3

Thank you!

I think I will set the delay time to 2 seconds then, should fit my requirements and provide good safety for read operation.

4

Would be more efficient to use refresh=wait_for option IMHO.

5

Well, I’m worried about this:

https://discuss.elastic.co/t/what-would-cause-refresh-wait-for-to-regularly-take-2-5-seconds/106369/8

I don’t mind the delay, but I can’t wait for a query for more than a second.

Arik

6

Wait for is used at index time.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.