After upgrade from 6.2.3 to 6.6.2 frequently gc overhead

[lysyjan87 (Jan Lysý)]

We have ES managed with Kubernetes and when we upgrade from 6.2.3 to 6.6.2 (or another higher minor version) then occur performance problems. In Cerebro is CPU and load on top. Some requests end with timeout.

Logs:

[2019-03-26T15:58:41,327][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][1236] overhead, spent [290ms] collecting in the last [1s]
[2019-03-26T15:59:04,373][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][1259] overhead, spent [288ms] collecting in the last [1s]
[2019-03-26T15:59:06,374][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][1261] overhead, spent [270ms] collecting in the last [1s]
[2019-03-26T16:04:54,480][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][1609] overhead, spent [267ms] collecting in the last [1s]
[2019-03-26T16:19:37,179][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][2491] overhead, spent [293ms] collecting in the last [1.1s]
[2019-03-26T16:27:04,313][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][2938] overhead, spent [318ms] collecting in the last [1s]
[2019-03-26T16:34:19,478][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][3373] overhead, spent [281ms] collecting in the last [1s]
[2019-03-26T16:42:18,615][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][3852] overhead, spent [267ms] collecting in the last [1s]
[2019-03-26T16:49:41,008][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][4294] overhead, spent [267ms] collecting in the last [1s]
[2019-03-26T17:00:00,195][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][4913] overhead, spent [253ms] collecting in the last [1s]

Part of ES stats:

{
  "_shards": {
    "total": 24,
    "successful": 24,
    "failed": 0
  },
  "_all": {
    "primaries": {
      "docs": {
        "count": 36410157,
        "deleted": 10640147
      },
      "store": {
        "size": "308.2gb",
        "size_in_bytes": 330975783719
      },
      "indexing": {
        "index_total": 14,
        "index_time": "676ms",
        "index_time_in_millis": 676,
        "index_current": 0,
        "index_failed": 0,
        "delete_total": 0,
        "delete_time": "0s",
        "delete_time_in_millis": 0,
        "delete_current": 0,
        "noop_update_total": 0,
        "is_throttled": false,
        "throttle_time": "0s",
        "throttle_time_in_millis": 0
    }
  }
}

Infrastructure details :
8 * r4.xlarge instances for ES
Volume type : gp2

Software details :
Elasticsearch Docker image : 6.6.2 (previous 6.2.3)

Can any one help me?