# Possible perf regression between 5.1.2 and 5.3.2? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 12, 2017, 8:57am UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995 "2017-06-12T08:57:24Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![dcausse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dcausse/32/97582_2.png) [@dcausse](https://discuss.elastic.co/u/dcausse)\ **Post date:** [June 12, 2017, 8:57am UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995/1 "2017-06-12T08:57:24Z")
Hi, we've just finished a rolling upgrade to bump from 5.1.2 to 5.3.2. We then noticed a visible perf regression in query percentiles [1](https://phabricator.wikimedia.org/T167636): - fulltext queries (p75): 33ms =\> 50ms - morelike (p75): 90ms =\> 150ms - comp suggest (p75): 10ms =\> 15ms Young gc activity increased while the overall heap usage seems to have decreased. We're still in the exploratory phase of this perf regression but any help would be welcome to narrow down our search scope. Thanks! ---
**Author:** ![Guillaume\_Lederrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guillaume_lederrey/32/18990_2.png) [@Guillaume\_Lederrey](https://discuss.elastic.co/u/Guillaume_Lederrey)\ **Post date:** [June 12, 2017, 9:22am UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995/2 "2017-06-12T09:22:12Z")
I'm working with @dcausse, I'm adding a few more details: [Cluster wide perf graphs](https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles) are on grafana. [Comparing individual nodes](https://grafana.wikimedia.org/dashboard/db/elasticsearch-node-comparison) is also possible (on the elastic1\* nodes). We upgraded the JDK to 1.8.0u131 (minor upgrade from 1.8.0u121). Comapring GC logs [before the upgrade](http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMTIvLS1nY19sb2QudGFyLmd6LS04LTU3LTQy) to [after the upgrade](http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTcvMDYvMTIvLS1nYy5sb2cudGFyLmd6LS04LTUyLTIy) for one of the servers, I can see: - allocation rate is fairly similar (from 1.44Gb/s to 1.35Gb/s) - avg young GC duration is similar (35ms) - young GC interval decreased from 6.8s to 1.1s =\> we are allocation about the same amount of heap, but taking much longer to collect it ---
**Author:** ![Guillaume\_Lederrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guillaume_lederrey/32/18990_2.png) [@Guillaume\_Lederrey](https://discuss.elastic.co/u/Guillaume_Lederrey)\ **Post date:** [June 12, 2017, 2:43pm UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995/3 "2017-06-12T14:43:55Z")
Digging into this, it looks like we re-aligned our JVM options on the standard elasticsearch. We activated Concurrent Mark and Sweep GC, which does [some strange things to NewRatio](https://bugs.openjdk.java.net/browse/JDK-8153578). Our young gen is now 2Gb instead of the previous 10Gb (the default NewRatio=2, with a total heap of 30Gb). I'm going to try remove that option and see if things go back to normal. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [June 12, 2017, 3:00pm UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995/4 "2017-06-12T15:00:01Z")
I remember playing with GC settings for Elasticsearch when I was with WMF but I didn't get anywhere as far as speed. I imagine those GC setting post-date me. It sounds like someone was able to get a significant bump by experimenting with them. I sure don't _think_ it was me. ---
**Author:** ![Guillaume\_Lederrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guillaume_lederrey/32/18990_2.png) [@Guillaume\_Lederrey](https://discuss.elastic.co/u/Guillaume_Lederrey)\ **Post date:** [June 12, 2017, 3:20pm UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995/5 "2017-06-12T15:20:37Z")
It wasn't me either 🙂 We still have some work to do to tune it better. In particular experimenting with Heap size. I'm pretty sure that our current 30Gb is way too large, but validating this is going to take some time... ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [June 12, 2017, 3:22pm UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995/6 "2017-06-12T15:22:12Z")
> [@Guillaume\_Lederrey](#): > > I'm pretty sure that our current 30Gb is way too large, but validating this is going to take some time... Indeed. Y'all don't use many of the things that take up a ton of heap. On 5.x you have many more protections against using too much heap too. ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 13, 2017, 7:53am UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995/7 "2017-06-13T07:53:33Z")
Would you mind sharing what your queries look like? Also,can you try to isolate the issue eg. by disabling highlighting, aggregations and simplifying the query as much as possible, and then adding those features back again one by one to try to see if there is anything in particular that makes your queries slower compared to 5.1.2? Capturing [nodes hot threads](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html) while the cluster is under load might also help. ---
**Author:** ![dcausse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dcausse/32/97582_2.png) [@dcausse](https://discuss.elastic.co/u/dcausse)\ **Post date:** [June 13, 2017, 8:50am UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995/8 "2017-06-13T08:50:11Z")
Sure, the queries we send are: - fulltext search: [json](https://en.wikipedia.org/w/index.php?search=foo%20bar&title=Special:Search&profile=default&fulltext=1&cirrusDumpQuery) (basically a filtered boolean + 2 rescore queries) - morelike: [json](https://en.wikipedia.org/w/index.php?search=morelike:Foo%20bar&title=Special:Search&profile=default&fulltext=1&cirrusDumpQuery) (a simple morelike query + 1 rescore query) - comp suggest: (sadly our query dump does not work for them) this is running 4 queries on 2 FST) Since the problem was detected on the production system it's hard for me to debug and isolate query components that may be slower. I'll setup a test environment and run some benchmarks. The hotthread does not show anything particular (mainly indexing threads). Thanks for your suggestions. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 11, 2017, 8:50am UTC](https://discuss.elastic.co/t/possible-perf-regression-between-5-1-2-and-5-3-2/88995/9 "2017-07-11T08:50:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.