ES master nodes CPU usage decrease over time and became red nodes

Hi People,

We have issues with our ElasticSearch cluster. Somehow, the CPU usage of the master nodes decreases over time (couple of days) and then crashes the cluster's state.

A screenshot of the CPU usage of Master 1:

We had to restart every node one-by-one to get the cluster up-and-running. We couldn't find related issues in es-cluster.log and in the gc.log file from that day, nothing different than common 'org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [finished_processing.response.Result] of type [float] in document with id 'zLJXtnMBAqpKShEiTlJF'
' issues.

Does someone know what process/issue could cause this issue? Did someone experienced it before?

Which version are you using? What is the size and specification of the cluster? How much data does it hold in terms of size on disk and shard count?

1 Like

Hi Christian,

We use version 6.8.10.

"nodes": {
    "count": {
      "total": 5,
      "data": 4,
      "coordinating_only": 1,
      "master": 3,
      "ingest": 4
    },
    "versions": [
      "6.8.10"
    ],
    "os": {
      "available_processors": 36,
      "allocated_processors": 36,
      "names": [
        {
          "name": "Linux",
          "count": 5
        }
      ],
"indices": {
    "count": 589,
    "shards": {
      "total": 5722,
      "primaries": 2861,
      "replication": 1,

We've got 589 indices and 5722 shards. A total of 1.8 TB data.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.