Index failed to prepare/warm -> Cause IO burst & high latency

Hi,

We are running a quite large cluster of ElasticSearch servers in
production, we see from time to time error like this;

[2013-03-25 08:47:16,860][WARN ][index.engine.robin ] [es37b]
[INDEX_NAME][178] failed to prepare/warm
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution of
[org.elasticsearch.indices.warmer.InternalIndicesWarmer$1]

always followed by a burst of IO, and the shard being not available (plus
the search queue being filled) for a time, because of the IO wait.
Basically, this cause my response times to go through the rough for some
time too..

My guess is that because of that, the FS cache is being invalidated and it
has to refill it from disk... I don't know if it's a bug or what can cause
this. My index is quite large (~9Billions documents, 250 shards, 0
replicas, ~20gb per shards), I am fully aware that with replicas the
problem should solve by itself, of at least being mitigated, but for now we
haven't enough IO to afford it. I am wondering if there is some way to at
least mitigate this issue for the time being.

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.