# Bad performance and crashes with Elasticsearch 5.1 **URL:** \ **Category:** Elasticsearch\ **Created:** [April 19, 2017, 11:37am UTC](https://discuss.elastic.co/t/bad-performance-and-crashes-with-elasticsearch-5-1/82875 "2017-04-19T11:37:24Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\ **Post date:** [April 19, 2017, 11:37am UTC](https://discuss.elastic.co/t/bad-performance-and-crashes-with-elasticsearch-5-1/82875/1 "2017-04-19T11:37:24Z")
We're running ELK 5.1.1; Elasticsearch runs on a standalone EC2 instance with 4CPUs/16GB RAM. We're indexing about 20GB daily, 40 indexes per day, so with 30-40 days retention we have ~800GB data, about 1800 indexes. This is staging environment, in our production we're running Elastic 2.x, and server with the same specs works nice with \>300GB/day, i.e. about 15 times more. As far as I can see, our traffic is very low for this server; our baseline is about 3% iowait and about 15% user CPU load. We have two issues: - all searches are slow. For instance, just basic discovery for the last week takes about 40 seconds. During that I see user CPU usage close to 100%, iowait stays low - up to 5%. Many queries are aborted by circuit breakers, in this case Elasticsearch stops indexing. - from time to time it stops indexing "jps -l -m -v" output: 16460 sun.tools.jps.Jps -l -m -v -Dapplication.home=/usr/lib/jvm/jdk1.8.0\_101 -Xms8m 1916 org.elasticsearch.bootstrap.Elasticsearch -d -p /var/run/elasticsearch/elasticsearch.pid -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/elasticsearch/data -Edefault.path.conf=/etc/elasticsearch -Xms8g -Xmx8g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Djna.tmpdir=/elasticsearch/tmp -Des.path.home=/usr/share/elasticsearch Any ideas? TIA, Vitaly ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [April 19, 2017, 12:20pm UTC](https://discuss.elastic.co/t/bad-performance-and-crashes-with-elasticsearch-5-1/82875/2 "2017-04-19T12:20:50Z")
All versions less than 5.2.2 are impacted by three issues: - the Netty recycler (Netty is the framework we use for networking) is fundamentally flawed, so we disabled it in 5.1.2 ([https://github.com/elastic/elasticsearch/pull/22452](https://github.com/elastic/elasticsearch/pull/22452)) - collecting stats while indexing could cause nodes to drop ([https://github.com/elastic/elasticsearch/pull/22317](https://github.com/elastic/elasticsearch/pull/22317)) - a client that disconnects during an HTTP request could cause a circuit breaker leak ([https://github.com/elastic/elasticsearch/pull/23310/files](https://github.com/elastic/elasticsearch/pull/23310/files)) From your description, I think you are most likely heavily impacted by the first issue. I think you should upgrade. ---
**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\ **Post date:** [April 26, 2017, 6:29am UTC](https://discuss.elastic.co/t/bad-performance-and-crashes-with-elasticsearch-5-1/82875/3 "2017-04-26T06:29:31Z")
Jason, many thanks! We upgraded to 5.3 last Thu, and so far system seems much better - no crashes during a week. Vitaly ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [April 26, 2017, 9:40am UTC](https://discuss.elastic.co/t/bad-performance-and-crashes-with-elasticsearch-5-1/82875/4 "2017-04-26T09:40:53Z")
You're welcome! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 24, 2017, 9:49am UTC](https://discuss.elastic.co/t/bad-performance-and-crashes-with-elasticsearch-5-1/82875/5 "2017-05-24T09:49:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.