# Performance of painless scripts in 6.x **URL:** \ **Category:** Elasticsearch\ **Created:** [June 25, 2018, 6:25pm UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308 "2018-06-25T18:25:26Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [June 25, 2018, 6:25pm UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/1 "2018-06-25T18:25:26Z")
I have tested with 6.3 + Java 10 in docker and also 6.2.4 + Java 8 in docker. Behavior is same. With the same query, ES 6.x latencies are 10x slower than 5.6.4. I have observed similar behavior with few other scripted queries also. What I have observed is that the heap usage is same in both, latency when there is no concurrency is same. When there is concurrency or continuous load even with 1 concurrency, ES 6.x CPU usage is very high(100% across all 4 cores). The only metric I see different is the young GC count which is very high in ES 6.x, which might explain the high CPU usage? Please refer the attached images. Is ES 6.x somehow less performant? it's hard to believe. So, is it something to do wth any default setting that has changed, that I need to tune? Data - 1.1 M docs 4 GB, 4 core with 2 GB set to heap ES query used to load test > **5.6.4** ![old3](https://us1.discourse-cdn.com/elastic/original/3X/3/0/30bc94980de36fa3eb624ca0eb69ac018c39fa36.png) ![old1](https://us1.discourse-cdn.com/elastic/original/3X/f/e/fe3831019b51ba2a5c46be8f08196997238a6d61.png) **6.2.4** ![new1|690x402] (upload://mdiKLWLrDo1ukGAzDOg0XZKinzJ.png) ![new3](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8bbce11c633a28f66b0eeb8869af67b6fe89a7db.png) ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [June 28, 2018, 5:34am UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/2 "2018-06-28T05:34:29Z")
Hi, hard to tell what is going on based on your description. Is there a chance that you can (privately) share an anonymised version of the document corpus so we can have a closer look? Daniel ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [June 28, 2018, 6:26am UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/3 "2018-06-28T06:26:34Z")
@danielmitterdorfer Thanks. Do you need the whole dataset + mapping or just few rows would be sufficient for you to test? ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [June 28, 2018, 6:44am UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/4 "2018-06-28T06:44:24Z")
The whole dataset (incl. mapping) would be great so we can reproduce your scenario. It would be great if you could share a download link via a private message. ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [June 28, 2018, 2:13pm UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/5 "2018-06-28T14:13:34Z")
Hi @danielmitterdorfer You were saying this > It would also be good if you could share the script with the actual field names from the data set because we suspect that the difference might have to do with doc values and then it is important again that we access the same fields that you do. Does that mean that the performance of doc\_values depends on the length of field names? Thanks. ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [June 29, 2018, 4:50am UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/6 "2018-06-29T04:50:14Z")
Hi, no, I was not referring to the length of the field names but it might make a difference whether you access a field that has 5 distinct values or 5 million distinct values. Daniel ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [June 29, 2018, 11:24am UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/7 "2018-06-29T11:24:40Z")
I created a benchmark based on the data that you have provided. It first indexes all data and then it runs your Painless query with four clients concurrently at a rate of five operations per second. I ran that against the Docker images for 5.6.4 and 6.2.4 with a heap size of 2GB on our [nightly benchmarking hardware](https://elasticsearch-benchmarks.elastic.co/). For that scenario I get the following results: | Metric | Task | ES 5.6.4 | ES 6.2.4 | Diff | Unit | | --- | --- | --- | --- | --- | --- | | Total Young Gen GC | | 2.974 | 2.431 | -0.543 | s | | Total Old Gen GC | | 0.211 | 0.181 | -0.03 | s | | Store size | | 0.514003 | 0.514522 | 0.00052 | GB | | 50th percentile latency | painless | 269.538 | 257.694 | -11.8438 | ms | | 90th percentile latency | painless | 302.744 | 290.283 | -12.4613 | ms | | 99th percentile latency | painless | 322.716 | 304.926 | -17.7899 | ms | | 100th percentile latency | painless | 367.526 | 317.751 | -49.7747 | ms | | error rate | painless | 0 | 0 | 0 | % | It could be the case that the benchmark is still not capturing your scenario or something else is going on in your environment. I'll send you a link to the benchmark in a PM so you can try it yourself and see whether you see a difference. ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [June 30, 2018, 1:45pm UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/8 "2018-06-30T13:45:13Z")
Thanks a lot @danielmitterdorfer I am gonna have to re run my tests and see if there is something different in my test approaches. ---
**Author:** ![vangap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vangap/32/44927_2.png) [@vangap](https://discuss.elastic.co/u/vangap)\ **Post date:** [July 2, 2018, 11:17am UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/9 "2018-07-02T11:17:11Z")
I have run my tests again. There was an issue with my load test where all requests were not reaching ES sometimes because of another issue. I have now run the queries directly on ES using apache bench and I do see similar latencies more or less between 5.6.x and 6.3. @danielmitterdorfer Thanks alot for your help. I should have looked at my test setup closely instead of suspecting the ES 🙂 ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [July 2, 2018, 11:50am UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/10 "2018-07-02T11:50:48Z")
Hi, thanks for the feedback. Glad it turned out that Elasticsearch performs equally well. 🙂 Daniel ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 30, 2018, 11:51am UTC](https://discuss.elastic.co/t/performance-of-painless-scripts-in-6-x/137308/11 "2018-07-30T11:51:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.