# Troubleshooting cluster wide performance slow downs

**URL:** <https://discuss.elastic.co/t/troubleshooting-cluster-wide-performance-slow-downs/17867>\
**Category:** Elasticsearch\
**Created:** [June 2, 2014, 10:18pm UTC](https://discuss.elastic.co/t/troubleshooting-cluster-wide-performance-slow-downs/17867 "2014-06-02T22:18:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jeff\_Moyer](https://avatars.discourse-cdn.com/v4/letter/j/ad7895/32.png) [@Jeff\_Moyer](https://discuss.elastic.co/u/Jeff_Moyer)\
**Post date:** [June 2, 2014, 10:18pm UTC](https://discuss.elastic.co/t/troubleshooting-cluster-wide-performance-slow-downs/17867/1 "2014-06-02T22:18:54Z")

</div>

Hello,

We have a 15 node cluster and have been experiencing intermittent cluster  
wide performance issues that last for between 1 and 5 minutes.

ES 1.1.0  
jdk1.7.0\_51

We started seeing this after upgrading from .0.9.3 to .1.1.0, although we  
have also made many other changes around the same time.

It seems to happen about every other day. We start logging slow queries  
(for queries that normally run 5ms to 20ms) with times in the several  
hundred ms range for 10 to 15 seconds on all nodes, then the performance  
drops to several seconds. Once this happens our open contexts climb from in  
the hundreds to over 300K. At this point all of our queues fill up and we  
log many  
"org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:  
rejected execution (queue capacity 1000)".

This usually lasts for 2-3 minutes and then everything recovers. Our  
clients time-out after 5 seconds and retry a couple of times contributing  
to the rapid queue build up.

This has been happening during our lowest query rate times, it has not  
occurred during our busiest times (which are many times greater query rate).

We have marvel installed and several other OS performance monitoring tools.

We do see a small CPU spike when this happens (from almost nothing to 10%  
on 12 cores), but no swapping or high GC activity. Looking through all of  
the marvel graphs, other than open contexts going through the roof and  
average query latency climbing, everything else looks normal and really low  
utilization compared to our busy times. No large merging, no increase in  
indexing activity, etc.

We have been through our application logs looking for for unusual query  
activity and I have captured wire data from every server on port 9200  
during an occurrence and combed through the incoming search activity and  
have not been able to find anything to correlate to the cluster behaviour.

Any advice on methods to troubleshoot this appreciated!

Thanks,  
Jeff

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d3f22167-7404-4db0-9231-513bf7cf2108%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d3f22167-7404-4db0-9231-513bf7cf2108%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:25am UTC](https://discuss.elastic.co/t/troubleshooting-cluster-wide-performance-slow-downs/17867/2 "2017-07-06T01:25:08Z")

</div>

Closed on July 6, 2017, 1:25am UTC