ES master nodes CPU usage decrease over time and became red nodes

Elastic Stack Elasticsearch
1

Hi People,

We have issues with our ElasticSearch cluster. Somehow, the CPU usage of the master nodes decreases over time (couple of days) and then crashes the cluster's state.

A screenshot of the CPU usage of Master 1:

Screenshot 2020-08-10 at 10.13.39
Screenshot 2020-08-10 at 10.13.391324×566 70.4 KB

We had to restart every node one-by-one to get the cluster up-and-running. We couldn't find related issues in es-cluster.log and in the gc.log file from that day, nothing different than common 'org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [finished_processing.response.Result] of type [float] in document with id 'zLJXtnMBAqpKShEiTlJF'
' issues.

Does someone know what process/issue could cause this issue? Did someone experienced it before?

2

Which version are you using? What is the size and specification of the cluster? How much data does it hold in terms of size on disk and shard count?

3

Hi Christian,

We use version 6.8.10.

"nodes": {
    "count": {
      "total": 5,
      "data": 4,
      "coordinating_only": 1,
      "master": 3,
      "ingest": 4
    },
    "versions": [
      "6.8.10"
    ],
    "os": {
      "available_processors": 36,
      "allocated_processors": 36,
      "names": [
        {
          "name": "Linux",
          "count": 5
        }
      ],
"indices": {
    "count": 589,
    "shards": {
      "total": 5722,
      "primaries": 2861,
      "replication": 1,

We've got 589 indices and 5722 shards. A total of 1.8 TB data.