I have tested with 6.3 + Java 10 in docker and also 6.2.4 + Java 8 in docker.
Behavior is same.
With the same query, ES 6.x latencies are 10x slower than 5.6.4. I have observed similar behavior with few other scripted queries also.
What I have observed is that the heap usage is same in both, latency when there is no concurrency is same.
When there is concurrency or continuous load even with 1 concurrency, ES 6.x CPU usage is very high(100% across all 4 cores).
The only metric I see different is the young GC count which is very high in ES 6.x, which might explain the high CPU usage? Please refer the attached images.
Is ES 6.x somehow less performant? it's hard to believe. So, is it something to do wth any default setting that has changed, that I need to tune?
Data - 1.1 M docs
4 GB, 4 core with 2 GB set to heap
ES query used to load test