Performance of painless scripts in 6.x

Elastic Stack Elasticsearch
1

I have tested with 6.3 + Java 10 in docker and also 6.2.4 + Java 8 in docker.
Behavior is same.

With the same query, ES 6.x latencies are 10x slower than 5.6.4. I have observed similar behavior with few other scripted queries also.

What I have observed is that the heap usage is same in both, latency when there is no concurrency is same.
When there is concurrency or continuous load even with 1 concurrency, ES 6.x CPU usage is very high(100% across all 4 cores).
The only metric I see different is the young GC count which is very high in ES 6.x, which might explain the high CPU usage? Please refer the attached images.
Is ES 6.x somehow less performant? it's hard to believe. So, is it something to do wth any default setting that has changed, that I need to tune?

Data - 1.1 M docs
4 GB, 4 core with 2 GB set to heap
ES query used to load test

5.6.4

old3
old3.png1611×781 203 KB
old1
old1.png1616×782 257 KB

6.2.4
![new1|690x402]
(upload://mdiKLWLrDo1ukGAzDOg0XZKinzJ.png)

new3
new3.png1620×877 167 KB

2

Hi,

hard to tell what is going on based on your description. Is there a chance that you can (privately) share an anonymised version of the document corpus so we can have a closer look?

Daniel

3

@danielmitterdorfer Thanks.
Do you need the whole dataset + mapping or just few rows would be sufficient for you to test?

4

The whole dataset (incl. mapping) would be great so we can reproduce your scenario. It would be great if you could share a download link via a private message.

5

Hi @danielmitterdorfer

You were saying this

It would also be good if you could share the script with the actual field names from the data set because we suspect that the difference might have to do with doc values and then it is important again that we access the same fields that you do.

Does that mean that the performance of doc_values depends on the length of field names?

Thanks.

6

Hi,

no, I was not referring to the length of the field names but it might make a difference whether you access a field that has 5 distinct values or 5 million distinct values.

Daniel

7

I created a benchmark based on the data that you have provided. It first indexes all data and then it runs your Painless query with four clients concurrently at a rate of five operations per second. I ran that against the Docker images for 5.6.4 and 6.2.4 with a heap size of 2GB on our nightly benchmarking hardware.

For that scenario I get the following results:

Metric Task ES 5.6.4 ES 6.2.4 Diff Unit
Total Young Gen GC 2.974 2.431 -0.543 s
Total Old Gen GC 0.211 0.181 -0.03 s
Store size 0.514003 0.514522 0.00052 GB
50th percentile latency painless 269.538 257.694 -11.8438 ms
90th percentile latency painless 302.744 290.283 -12.4613 ms
99th percentile latency painless 322.716 304.926 -17.7899 ms
100th percentile latency painless 367.526 317.751 -49.7747 ms
error rate painless 0 0 0 %

It could be the case that the benchmark is still not capturing your scenario or something else is going on in your environment. I'll send you a link to the benchmark in a PM so you can try it yourself and see whether you see a difference.

1 Like
8

Thanks a lot @danielmitterdorfer
I am gonna have to re run my tests and see if there is something different in my test approaches.

9

I have run my tests again.
There was an issue with my load test where all requests were not reaching ES sometimes because of another issue.

I have now run the queries directly on ES using apache bench and I do see similar latencies more or less between 5.6.x and 6.3.

@danielmitterdorfer Thanks alot for your help. I should have looked at my test setup closely instead of suspecting the ES :slight_smile:

10

Hi,

thanks for the feedback. Glad it turned out that Elasticsearch performs equally well. :slight_smile:

Daniel

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.