While shard was relocating, experienced very high search response times

We use rolling daily indexes. After we deleted one, Elasticsearch decided
to relocate a replica shard to another node. Totally fine, that's what I
expect Elasticsearch to do.

What I didn't expect was for search queries that might have used that shard
to bomb out. Writes to different indexes still performed fine, but searches
either timed out or returned with very long response times. Once the shard
completed relocating (15min for a 26gb shard), performance returned to
normal.

We have all the logs and marvel data for this time period, and I don't see
anything that seems out of the ordinary. What logs/settings I should be
looking at so we don't have this problem again?

Thanks,

--
Nick Canzoneri
Developer, Wildbit http://wildbit.com/
Beanstalk http://beanstalkapp.com/, Postmark http://postmarkapp.com/,
dploy.io

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKWm5yPfQ1dLW-QengDs6gxfP9bjhnH7q7qnR-kYnY7nZu8d_g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.