Hello,
We have a 15 node cluster and have been experiencing intermittent cluster
wide performance issues that last for between 1 and 5 minutes.
ES 1.1.0
jdk1.7.0_51
We started seeing this after upgrading from .0.9.3 to .1.1.0, although we
have also made many other changes around the same time.
It seems to happen about every other day. We start logging slow queries
(for queries that normally run 5ms to 20ms) with times in the several
hundred ms range for 10 to 15 seconds on all nodes, then the performance
drops to several seconds. Once this happens our open contexts climb from in
the hundreds to over 300K. At this point all of our queues fill up and we
log many
"org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 1000)".
This usually lasts for 2-3 minutes and then everything recovers. Our
clients time-out after 5 seconds and retry a couple of times contributing
to the rapid queue build up.
This has been happening during our lowest query rate times, it has not
occurred during our busiest times (which are many times greater query rate).
We have marvel installed and several other OS performance monitoring tools.
We do see a small CPU spike when this happens (from almost nothing to 10%
on 12 cores), but no swapping or high GC activity. Looking through all of
the marvel graphs, other than open contexts going through the roof and
average query latency climbing, everything else looks normal and really low
utilization compared to our busy times. No large merging, no increase in
indexing activity, etc.
We have been through our application logs looking for for unusual query
activity and I have captured wire data from every server on port 9200
during an occurrence and combed through the incoming search activity and
have not been able to find anything to correlate to the cluster behaviour.
Any advice on methods to troubleshoot this appreciated!
Thanks,
Jeff
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d3f22167-7404-4db0-9231-513bf7cf2108%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.