Refresh latency

[manast (Michalis A Anastasopoulos)]

We are experiencing significant latency in the refresh operation. In some cases refresh takes up to one minute to complete.

Our index is about 150GB large and has 32 shards spread across 12 nodes.

The refresh interval is currently set to 15s.

My questions:

• Would a lower refresh interval speed up refresh at the cost of lower indexing speed?
• Would more nodes help to increase refresh performance?
• Are there any other measures to be done for increasing refresh performance?

Thanks,
Michail

[warkolm (Mark Walkom)]

@manast

Would a lower refresh interval speed up refresh at the cost of lower indexing speed?
Would more nodes help to increase refresh performance?

Possibly, to both.

What version are you on?
What sort of infrastructure?

[manast (Michalis A Anastasopoulos)]

We are using Elasticsearch 2.2.0.
We have 12 data nodes and 3 master nodes.
The data nodes have 24 GB RAM and 300 GB SSD capacity each.

[warkolm (Mark Walkom)]

What OS, what IO scheduler?

[manast (Michalis A Anastasopoulos)]

We have CentOS 7.3.1611
with the deadline scheduler

[warkolm (Mark Walkom)]

Try the noop scheduler and see if that helps.

[manast (Michalis A Anastasopoulos)]

It turns out that we don't have only SSDs. Our cluster is connected to various types of NAS including flash, SSDs and HDDs. Do you think that the NOOP scheduler would still help?