After upgrade from 6.2.3 to 6.6.2 frequently gc overhead

Elastic Stack Elasticsearch
1

We have ES managed with Kubernetes and when we upgrade from 6.2.3 to 6.6.2 (or another higher minor version) then occur performance problems. In Cerebro is CPU and load on top. Some requests end with timeout.

Logs:

[2019-03-26T15:58:41,327][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][1236] overhead, spent [290ms] collecting in the last [1s]
[2019-03-26T15:59:04,373][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][1259] overhead, spent [288ms] collecting in the last [1s]
[2019-03-26T15:59:06,374][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][1261] overhead, spent [270ms] collecting in the last [1s]
[2019-03-26T16:04:54,480][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][1609] overhead, spent [267ms] collecting in the last [1s]
[2019-03-26T16:19:37,179][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][2491] overhead, spent [293ms] collecting in the last [1.1s]
[2019-03-26T16:27:04,313][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][2938] overhead, spent [318ms] collecting in the last [1s]
[2019-03-26T16:34:19,478][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][3373] overhead, spent [281ms] collecting in the last [1s]
[2019-03-26T16:42:18,615][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][3852] overhead, spent [267ms] collecting in the last [1s]
[2019-03-26T16:49:41,008][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][4294] overhead, spent [267ms] collecting in the last [1s]
[2019-03-26T17:00:00,195][INFO ][o.e.m.j.JvmGcMonitorService] [data-5cw6k] [gc][4913] overhead, spent [253ms] collecting in the last [1s]

Part of ES stats:

{
  "_shards": {
    "total": 24,
    "successful": 24,
    "failed": 0
  },
  "_all": {
    "primaries": {
      "docs": {
        "count": 36410157,
        "deleted": 10640147
      },
      "store": {
        "size": "308.2gb",
        "size_in_bytes": 330975783719
      },
      "indexing": {
        "index_total": 14,
        "index_time": "676ms",
        "index_time_in_millis": 676,
        "index_current": 0,
        "index_failed": 0,
        "delete_total": 0,
        "delete_time": "0s",
        "delete_time_in_millis": 0,
        "delete_current": 0,
        "noop_update_total": 0,
        "is_throttled": false,
        "throttle_time": "0s",
        "throttle_time_in_millis": 0
    }
  }
}

Infrastructure details :
8 * r4.xlarge instances for ES
Volume type : gp2

Software details :
Elasticsearch Docker image : 6.6.2 (previous 6.2.3)

Can any one help me?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.