Refresh_interval less than 1 per second?

Elastic Stack Elasticsearch
1

Hi guys,

We are syncing data from Couchbase into Elasticsearch via CBES connector and have up to ~550ms delay from CBES inserting data into ES.

Afterwards it takes up to 2 seconds for data to become searchable in ES.

Do I understand correctly that there is unavoidable delay of up to 1 second from the refresh_interval index setting? Can refresh_interval be set to less than 1 second?

The goal is to decrease latency between data being inserted in Couchbase and appearing for search in Elasticsearch.

2

I've applied 200ms refresh_interval to an index in test env. Are we going to have any adverse effects in a big indexes? How low can it go realistically?

3

I don't think is there is a lower limit, but the refresh interval can have a huge impact in the indexing performance and in the overall performance.

For example, when you have indexing performance issues, one of the recommendations is to increase the refresh interval.

So yes, using small refresh interval can have adverse effects, I would say that anything lower than the default 1s would not be recommended.

But you will need to try it out and see if it works for you.

4

Thank you Leandro,

Do you know someone running refresh_interval lower than 1/s in Production? I'll start lowering it by 100-200ms increments and see if load is acceptable.

Can you suggest which parameters should I monitor after decreasing this parameter? To check for any adverse effects.

5

A big +1 to this.

6

In addition to altering the refresh interval you can also set a parameter per request to refresh once the request has completed. This however genarally has a huge impact on indexing throughpout as it results in a lot of additional processing, disk I/O and a lot of very small segments that need to be merged. I assume this would also negatively affect search performance. I would expect setting a very short refresh interval to have similar effects.

7

And only way to assess negative performance impact is by trial? If I have some resources to spare should it work reliably and only adverse effect is performance degradation?

8

I do not know if that is the only downside or not.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.