Hi, we've migrated our production server cluster (1 master node, 3 data node, 2TB data) from 5.6 to 6.8.5 by using the rolling-update procedure (https://www.elastic.co/guide/en/elasticsearch/reference/6.8/rolling-upgrades.html).
Previously we did a rolling upgrade from 5.4 to 5.6: all went fine, no performance issue.
Although we have the same infrastructure (os,network,memory, disk) and application profile (same input and output frequencies) the platform with 6.8.5 now shows a huge decrease of performance.
We've tried disabling all the x-pack module (we do not need them - apart monitoring).
Data nodes continuously performs garbage collection every second, producing output like this:
[gc][153041] overhead, spent [361ms] collecting in the last [1s].
Search and index times are more than doubled with respect to previous version.
We tried to change the garbage collector from G1GC to CMS (we have jvm 1.8.0_191), but we had no positive change.
It is not clear to me why having not changed anything (apart elastic release) we're encountering such deep performance issues.
As well as it is not clear if the data nodes in addition to perform search/index as required by our application, are also converting the content of the indexes from release 5.6 to 6.8.
Thanks in advance for any support.
/gio