Refresh time/latency

Elastic Stack Elasticsearch
1

Hi,

I've been reading https://www.elastic.co/guide/en/elasticsearch/guide/current/near-real-time.html and it seems to imply that an index refresh is guaranteed to occur within 1 second. Is this true regardless of the size of the index, indexing throughput, GC, disk IO (we're running on EC2 instances with EBS)?

Does anyone have any p50, p90, p100 metrics for large, high index throughput indexes?

Appreciate the help.

2

There is a refresh thread pool and each index has a job in that pool. When one refresh finishes the job is rescheduled for a second later. It looks to me like the maximum will always be a few milliseconds longer than a second. I suspect the average will be right around a half second though. Higher throughput is going to increase those few milliseconds. As will slow disks. As will GC, but if GC is a problem you have bigger problems. The size of the index should be irrelevant except in that larger indexes might have more going on like merges and searches that can cause disk IO or CPU usage.

That all being said I haven't heard folks having trouble with refreshes falling behind. What falls behind on slow disks when you push a high throughput is merges and the way to help that is to increase the refresh interval so you make fewer small segments. Its not perfect, but its an easy nob to tune.

Elasticsearch has a backpressure mechanism to slow down indexing throughput if merges fall too far behind so you might notice that indexing itself just slows down. Elasticsearch will log something about that when it happens. And the client's won't get acknowledgements to their indexing requests are finished so they will feel the backpressure.

1 Like