Collector [cluster_stats] timed out when collecting data: node

Hello everyone,

I have a problem if someone can help me it would be very appreciated.
For several weeks I have noticed that my elastic cluster is unavailable for about 1 hour almost all the time at the same time on Sunday between 1am and 2am. I investigated and I came out that I must have too many shards in my cluster compared to its size so I reindexed a good part of the smaller indexes in order to solve the problem.
I now adhere to the best practices of 20 times the gigabyte heap of rubber band memory. However, I had the same problem again last Sunday and I don't understand the cause. I can attach parameters and stats of my cluster if needed.

i'm using elastick satck 8.3.2

4 node :

  • 3 HOT / WARM
  • 1 COLD

Cluster :
Primary Shards : 80
Replica Shards : 74

{"log":"{\"@timestamp\":\"2022-11-27T01:00:27.627Z\", \"log.level\":\"ERROR\", \"message\":\"collector [cluster_stats] timed out when collecting data: node [AMjWTf-GQYyqgoZb2q4wKg] did not respond within [10s]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1126]\",\"log.logger\":\"org.elasticsearch.xpack.monitoring.collector.cluster.ClusterStatsCollector\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:00:27.627731148Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:32.572Z\", \"log.level\": \"WARN\", \"message\":\"failed to retrieve stats for node [AMjWTf-GQYyqgoZb2q4wKg]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1156]\",\"log.logger\":\"org.elasticsearch.cluster.InternalClusterInfoService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\",\"error.type\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException\",\"error.message\":\"[es02][10.0.30.12:9500][cluster:monitor/nodes/stats[n]] request_id [239966419] timed out after [15010ms]\",\"error.stack_trace\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException: [es02][10.0.30.12:9500][cluster:monitor/nodes/stats[n]] request_id [239966419] timed out after [15010ms]\\n\"}\n","stream":"stdout","time":"2022-11-27T01:00:32.573130191Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:32.575Z\", \"log.level\": \"WARN\", \"message\":\"failed to retrieve shard stats from node [AMjWTf-GQYyqgoZb2q4wKg]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1142]\",\"log.logger\":\"org.elasticsearch.cluster.InternalClusterInfoService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\",\"error.type\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException\",\"error.message\":\"[es02][10.0.30.12:9500][indices:monitor/stats[n]] request_id [239966424] timed out after [15010ms]\",\"error.stack_trace\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException: [es02][10.0.30.12:9500][indices:monitor/stats[n]] request_id [239966424] timed out after [15010ms]\\n\"}\n","stream":"stdout","time":"2022-11-27T01:00:32.575670636Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:37.628Z\", \"log.level\":\"ERROR\", \"message\":\"collector [enrich_coordinator_stats] timed out when collecting data: java.util.concurrent.TimeoutException: Timeout waiting for task.\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1126]\",\"log.logger\":\"org.elasticsearch.xpack.monitoring.collector.enrich.EnrichStatsCollector\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:00:37.628367826Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:44.583Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [27s/27018ms] ago, timed out [17s/17012ms] ago, action [internal:coordination/fault_detection/follower_check], node [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}{rack=r0, ml.machine_memory=6442450944, xpack.installed=true, ml.max_jvm_size=3221225472}], id [239966413]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#2]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:00:44.583571218Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:44.583Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [16s/16011ms] ago, timed out [6s/6003ms] ago, action [internal:coordination/fault_detection/follower_check], node [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}{rack=r0, ml.machine_memory=6442450944, xpack.installed=true, ml.max_jvm_size=3221225472}], id [239966484]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#2]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:00:47.525257451Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:44.640Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [27s/27018ms] ago, timed out [17s/17012ms] ago, action [cluster:monitor/stats[n]], node [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}{rack=r0, ml.machine_memory=6442450944, xpack.installed=true, ml.max_jvm_size=3221225472}], id [239966432]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#1]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:00:47.525282252Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:44.640Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [27s/27018ms] ago, timed out [12s/12008ms] ago, action [cluster:monitor/nodes/stats[n]], node [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}{rack=r0, ml.machine_memory=6442450944, xpack.installed=true, ml.max_jvm_size=3221225472}], id [239966419]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#1]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:00:47.525290152Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:44.641Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [27s/27018ms] ago, timed out [12s/12008ms] ago, action [indices:monitor/stats[n]], node [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}{rack=r0, ml.machine_memory=6442450944, xpack.installed=true, ml.max_jvm_size=3221225472}], id [239966424]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#2]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:00:47.525296052Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:47.630Z\", \"log.level\":\"ERROR\", \"message\":\"collector [index_recovery] timed out when collecting data: node [AMjWTf-GQYyqgoZb2q4wKg] did not respond within [10s]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1126]\",\"log.logger\":\"org.elasticsearch.xpack.monitoring.collector.indices.IndexRecoveryCollector\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:00:47.631317845Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:00:57.632Z\", \"log.level\":\"ERROR\", \"message\":\"collector [index-stats] timed out when collecting data: node [AMjWTf-GQYyqgoZb2q4wKg] did not respond within [10s]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1126]\",\"log.logger\":\"org.elasticsearch.xpack.monitoring.collector.indices.IndexStatsCollector\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:00:57.632420158Z"}

Thanks in advance

What is the output from the _cluster/stats?pretty&human API?

Is there more to your logs that you can share?

{
  "_nodes": {
    "total": 4,
    "successful": 4,
    "failed": 0
  },
  "cluster_name": "ds-monitoring-prod",
  "cluster_uuid": "o3uSGc7TR0O503-YKA9kLQ",
  "timestamp": 1669709889200,
  "status": "green",
  "indices": {
    "count": 72,
    "shards": {
      "total": 124,
      "primaries": 73,
      "replication": 0.6986301369863014,
      "index": {
        "shards": {
          "min": 1,
          "max": 4,
          "avg": 1.7222222222222223
        },
        "primaries": {
          "min": 1,
          "max": 2,
          "avg": 1.0138888888888888
        },
        "replication": {
          "min": 0,
          "max": 2,
          "avg": 0.6944444444444444
        }
      }
    },
    "docs": {
      "count": 306603362,
      "deleted": 3083168
    },
    "store": {
      "size": "185.8gb",
      "size_in_bytes": 199597870539,
      "total_data_set_size": "185.8gb",
      "total_data_set_size_in_bytes": 199597870539,
      "reserved": "0b",
      "reserved_in_bytes": 0
    },
    "fielddata": {
      "memory_size": "4.8mb",
      "memory_size_in_bytes": 5036120,
      "evictions": 0
    },
    "query_cache": {
      "memory_size": "30.6mb",
      "memory_size_in_bytes": 32105041,
      "total_count": 137820852,
      "hit_count": 6729641,
      "miss_count": 131091211,
      "cache_size": 19377,
      "cache_count": 131775,
      "evictions": 112398
    },
    "completion": {
      "size": "0b",
      "size_in_bytes": 0
    },
    "segments": {
      "count": 936,
      "memory": "0b",
      "memory_in_bytes": 0,
      "terms_memory": "0b",
      "terms_memory_in_bytes": 0,
      "stored_fields_memory": "0b",
      "stored_fields_memory_in_bytes": 0,
      "term_vectors_memory": "0b",
      "term_vectors_memory_in_bytes": 0,
      "norms_memory": "0b",
      "norms_memory_in_bytes": 0,
      "points_memory": "0b",
      "points_memory_in_bytes": 0,
      "doc_values_memory": "0b",
      "doc_values_memory_in_bytes": 0,
      "index_writer_memory": "18.2mb",
      "index_writer_memory_in_bytes": 19097380,
      "version_map_memory": "29.8kb",
      "version_map_memory_in_bytes": 30576,
      "fixed_bit_set": "41.9mb",
      "fixed_bit_set_memory_in_bytes": 43967624,
      "max_unsafe_auto_id_timestamp": 1669680003587,
      "file_sizes": {}
    },
    "mappings": {
      "field_types": [
        {
          "name": "alias",
          "count": 2689,
          "index_count": 17,
          "script_count": 0
        },
        {
          "name": "binary",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "boolean",
          "count": 1371,
          "index_count": 47,
          "script_count": 0
        },
        {
          "name": "byte",
          "count": 7,
          "index_count": 7,
          "script_count": 0
        },
        {
          "name": "constant_keyword",
          "count": 43,
          "index_count": 15,
          "script_count": 0
        },
        {
          "name": "date",
          "count": 1736,
          "index_count": 58,
          "script_count": 0
        },
        {
          "name": "date_range",
          "count": 1,
          "index_count": 1,
          "script_count": 0
        },
        {
          "name": "double",
          "count": 2466,
          "index_count": 15,
          "script_count": 0
        },
        {
          "name": "flattened",
          "count": 234,
          "index_count": 11,
          "script_count": 0
        },
        {
          "name": "float",
          "count": 2079,
          "index_count": 35,
          "script_count": 0
        },
        {
          "name": "geo_point",
          "count": 150,
          "index_count": 16,
          "script_count": 0
        },
        {
          "name": "half_float",
          "count": 94,
          "index_count": 26,
          "script_count": 0
        },
        {
          "name": "integer",
          "count": 157,
          "index_count": 8,
          "script_count": 0
        },
        {
          "name": "ip",
          "count": 911,
          "index_count": 20,
          "script_count": 0
        },
        {
          "name": "keyword",
          "count": 33967,
          "index_count": 58,
          "script_count": 0
        },
        {
          "name": "long",
          "count": 31584,
          "index_count": 48,
          "script_count": 0
        },
        {
          "name": "match_only_text",
          "count": 583,
          "index_count": 11,
          "script_count": 0
        },
        {
          "name": "nested",
          "count": 209,
          "index_count": 22,
          "script_count": 0
        },
        {
          "name": "object",
          "count": 30356,
          "index_count": 54,
          "script_count": 0
        },
        {
          "name": "scaled_float",
          "count": 1076,
          "index_count": 11,
          "script_count": 0
        },
        {
          "name": "short",
          "count": 1127,
          "index_count": 15,
          "script_count": 0
        },
        {
          "name": "text",
          "count": 2894,
          "index_count": 51,
          "script_count": 0
        },
        {
          "name": "version",
          "count": 4,
          "index_count": 4,
          "script_count": 0
        },
        {
          "name": "wildcard",
          "count": 187,
          "index_count": 11,
          "script_count": 0
        }
      ],
      "runtime_field_types": []
    },
    "analysis": {
      "char_filter_types": [],
      "tokenizer_types": [],
      "filter_types": [],
      "analyzer_types": [],
      "built_in_char_filters": [],
      "built_in_tokenizers": [],
      "built_in_filters": [],
      "built_in_analyzers": [
        {
          "name": "simple",
          "count": 2,
          "index_count": 1
        }
      ]
    },
    "versions": [
      {
        "version": "7.3.2",
        "index_count": 7,
        "primary_shard_count": 7,
        "total_primary_size": "174.5mb",
        "total_primary_bytes": 183055645
      },
      {
        "version": "7.17.3",
        "index_count": 6,
        "primary_shard_count": 6,
        "total_primary_size": "42.2mb",
        "total_primary_bytes": 44347082
      },
      {
        "version": "8.0.1",
        "index_count": 3,
        "primary_shard_count": 3,
        "total_primary_size": "153.4kb",
        "total_primary_bytes": 157159
      },
      {
        "version": "8.3.2",
        "index_count": 56,
        "primary_shard_count": 57,
        "total_primary_size": "164.3gb",
        "total_primary_bytes": 176485648568
      }
    ]
  },
  "nodes": {
    "count": {
      "total": 4,
      "coordinating_only": 0,
      "data": 3,
      "data_cold": 1,
      "data_content": 3,
      "data_frozen": 0,
      "data_hot": 3,
      "data_warm": 3,
      "ingest": 3,
      "master": 4,
      "ml": 3,
      "remote_cluster_client": 0,
      "transform": 3,
      "voting_only": 1
    },
    "versions": [
      "8.3.2"
    ],
    "os": {
      "available_processors": 7,
      "allocated_processors": 7,
      "names": [
        {
          "name": "Linux",
          "count": 4
        }
      ],
      "pretty_names": [
        {
          "pretty_name": "Ubuntu 20.04.4 LTS",
          "count": 4
        }
      ],
      "architectures": [
        {
          "arch": "amd64",
          "count": 4
        }
      ],
      "mem": {
        "total": "24gb",
        "total_in_bytes": 25769803776,
        "adjusted_total": "24gb",
        "adjusted_total_in_bytes": 25769803776,
        "free": "1gb",
        "free_in_bytes": 1122377728,
        "used": "22.9gb",
        "used_in_bytes": 24647426048,
        "free_percent": 4,
        "used_percent": 96
      }
    },
    "process": {
      "cpu": {
        "percent": 9
      },
      "open_file_descriptors": {
        "min": 554,
        "max": 944,
        "avg": 800
      }
    },
    "jvm": {
      "max_uptime": "94.5d",
      "max_uptime_in_millis": 8170884297,
      "versions": [
        {
          "version": "18.0.1.1",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "vm_version": "18.0.1.1+2-6",
          "vm_vendor": "Oracle Corporation",
          "bundled_jdk": true,
          "using_bundled_jdk": true,
          "count": 4
        }
      ],
      "mem": {
        "heap_used": "6.7gb",
        "heap_used_in_bytes": 7284386896,
        "heap_max": "12gb",
        "heap_max_in_bytes": 12884901888
      },
      "threads": 303
    },
    "fs": {
      "total": "511.2gb",
      "total_in_bytes": 548992692224,
      "free": "354.3gb",
      "free_in_bytes": 380506648576,
      "available": "354.3gb",
      "available_in_bytes": 380506648576
    },
    "plugins": [],
    "network_types": {
      "transport_types": {
        "security4": 4
      },
      "http_types": {
        "security4": 4
      }
    },
    "discovery_types": {
      "multi-node": 4
    },
    "packaging_types": [
      {
        "flavor": "default",
        "type": "docker",
        "count": 4
      }
    ],
    "ingest": {
      "number_of_pipelines": 23,
      "processor_stats": {
        "conditional": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "convert": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "geoip": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "grok": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "gsub": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "remove": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "rename": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "script": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "set": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        },
        "set_security_user": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time": "0s",
          "time_in_millis": 0
        }
      }
    },
    "indexing_pressure": {
      "memory": {
        "current": {
          "combined_coordinating_and_primary": "0b",
          "combined_coordinating_and_primary_in_bytes": 0,
          "coordinating": "0b",
          "coordinating_in_bytes": 0,
          "primary": "0b",
          "primary_in_bytes": 0,
          "replica": "0b",
          "replica_in_bytes": 0,
          "all": "0b",
          "all_in_bytes": 0
        },
        "total": {
          "combined_coordinating_and_primary": "0b",
          "combined_coordinating_and_primary_in_bytes": 0,
          "coordinating": "0b",
          "coordinating_in_bytes": 0,
          "primary": "0b",
          "primary_in_bytes": 0,
          "replica": "0b",
          "replica_in_bytes": 0,
          "all": "0b",
          "all_in_bytes": 0,
          "coordinating_rejections": 0,
          "primary_rejections": 0,
          "replica_rejections": 0
        },
        "limit": "0b",
        "limit_in_bytes": 0
      }
    }
  }
}

more log here only on master node when the event is produce

{"log":"{\"@timestamp\":\"2022-11-27T01:01:17.576Z\", \"log.level\": \"WARN\", \"message\":\"failed to retrieve stats for node [u0DuBr2mTH-d3fxDJbbhXg]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1063]\",\"log.logger\":\"org.elasticsearch.cluster.InternalClusterInfoService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\",\"error.type\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException\",\"error.message\":\"[es01][10.0.30.12:9300][cluster:monitor/nodes/stats[n]] request_id [239966627] timed out after [15010ms]\",\"error.stack_trace\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException: [es01][10.0.30.12:9300][cluster:monitor/nodes/stats[n]] request_id [239966627] timed out after [15010ms]\\n\"}\n","stream":"stdout","time":"2022-11-27T01:01:17.57667202Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:23.280Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [27.6s/27618ms] ago, timed out [17.8s/17811ms] ago, action [internal:coordination/fault_detection/follower_check], node [{es01}{u0DuBr2mTH-d3fxDJbbhXg}{h8GVRQ0ATf6UtONWcFYVWw}{es01}{10.0.30.12}{10.0.30.12:9300}{dhilmstw}{ml.machine_memory=6442450944, rack=r0, ml.max_jvm_size=3221225472, xpack.installed=true}], id [239966613]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#2]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:23.281136862Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:23.280Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [16.8s/16811ms] ago, timed out [6.8s/6804ms] ago, action [internal:coordination/fault_detection/follower_check], node [{es01}{u0DuBr2mTH-d3fxDJbbhXg}{h8GVRQ0ATf6UtONWcFYVWw}{es01}{10.0.30.12}{10.0.30.12:9300}{dhilmstw}{ml.machine_memory=6442450944, rack=r0, ml.max_jvm_size=3221225472, xpack.installed=true}], id [239966638]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#2]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:23.281191263Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:23.280Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [20.6s/20613ms] ago, timed out [5.6s/5603ms] ago, action [cluster:monitor/nodes/stats[n]], node [{es01}{u0DuBr2mTH-d3fxDJbbhXg}{h8GVRQ0ATf6UtONWcFYVWw}{es01}{10.0.30.12}{10.0.30.12:9300}{dhilmstw}{ml.machine_memory=6442450944, rack=r0, ml.max_jvm_size=3221225472, xpack.installed=true}], id [239966627]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#2]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:23.281217164Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:23.358Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [45.6s/45629ms] ago, timed out [35.6s/35623ms] ago, action [indices:monitor/recovery[n]], node [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}{rack=r0, ml.machine_memory=6442450944, xpack.installed=true, ml.max_jvm_size=3221225472}], id [239966539]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#1]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:23.358643445Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:23.359Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [35.6s/35623ms] ago, timed out [25.6s/25616ms] ago, action [indices:monitor/stats[n]], node [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}{rack=r0, ml.machine_memory=6442450944, xpack.installed=true, ml.max_jvm_size=3221225472}], id [239966565]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#1]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:23.359534061Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:23.407Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [32s/32020ms] ago, timed out [22s/22014ms] ago, action [internal:coordination/fault_detection/follower_check], node [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}{rack=r0, ml.machine_memory=6442450944, xpack.installed=true, ml.max_jvm_size=3221225472}], id [239966591]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#2]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:23.407914425Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:23.407Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [21s/21013ms] ago, timed out [11s/11007ms] ago, action [internal:coordination/fault_detection/follower_check], node [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}{rack=r0, ml.machine_memory=6442450944, xpack.installed=true, ml.max_jvm_size=3221225472}], id [239966625]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#2]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:23.408116828Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:23.411Z\", \"log.level\": \"INFO\",  \"current.health\":\"RED\",\"message\":\"Cluster health status changed from [GREEN] to [RED] (reason: [{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw} reason: followers check retry count exceeded [timeouts=2, failures=1]]).\",\"previous.health\":\"GREEN\",\"reason\":\"{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw} reason: followers check retry count exceeded [timeouts=2, failures=1]\" , \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][masterService#updateTask][T#1]\",\"log.logger\":\"org.elasticsearch.cluster.routing.allocation.AllocationService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:23.411917596Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:23.437Z\", \"log.level\": \"INFO\", \"message\":\"node-left[{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw} reason: followers check retry count exceeded [timeouts=2, failures=1]], term: 207, version: 25928, delta: removed {{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}}\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][masterService#updateTask][T#1]\",\"log.logger\":\"org.elasticsearch.cluster.service.MasterService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:23.438260166Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:33.446Z\", \"log.level\": \"INFO\", \"message\":\"after [10s] publication of cluster state version [25928] is still waiting for {es01}{u0DuBr2mTH-d3fxDJbbhXg}{h8GVRQ0ATf6UtONWcFYVWw}{es01}{10.0.30.12}{10.0.30.12:9300}{dhilmstw}{ml.machine_memory=6442450944, rack=r0, ml.max_jvm_size=3221225472, xpack.installed=true} [SENT_PUBLISH_REQUEST], {es03}{ElsMpBDUTa2m8LnuH-Boyg}{OTs8MAdwTz2vdO0zThOCwQ}{es03}{10.0.30.10}{10.0.30.10:9700}{dhilmstw}{xpack.installed=true, rack=r0, ml.machine_memory=6442450944, ml.max_jvm_size=3221225472} [WAITING_FOR_QUORUM], {es04}{6gJR1cKKRQWr8AruXLiDhg}{AcfjK4fVRYC-DQEZG97q_w}{es04}{10.0.30.11}{10.0.30.11:9900}{cmv}{rack=r1, xpack.installed=true} [SENT_PUBLISH_REQUEST]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][cluster_coordination][T#1]\",\"log.logger\":\"org.elasticsearch.cluster.coordination.Coordinator.CoordinatorPublication\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:33.447022005Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:01:53.446Z\", \"log.level\": \"INFO\", \"message\":\"removed {{es02}{AMjWTf-GQYyqgoZb2q4wKg}{tYRjQaIFQi6GY_jmFXQspA}{es02}{10.0.30.12}{10.0.30.12:9500}{dhilmstw}}, term: 207, version: 25928, reason: Publication{term=207, version=25928}\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][clusterApplierService#updateTask][T#1]\",\"log.logger\":\"org.elasticsearch.cluster.service.ClusterApplierService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:01:53.446950752Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:02:02.579Z\", \"log.level\": \"WARN\", \"message\":\"failed to retrieve stats for node [u0DuBr2mTH-d3fxDJbbhXg]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#930]\",\"log.logger\":\"org.elasticsearch.cluster.InternalClusterInfoService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\",\"error.type\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException\",\"error.message\":\"[es01][10.0.30.12:9300][cluster:monitor/nodes/stats[n]] request_id [239967008] timed out after [14811ms]\",\"error.stack_trace\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException: [es01][10.0.30.12:9300][cluster:monitor/nodes/stats[n]] request_id [239967008] timed out after [14811ms]\\n\"}\n","stream":"stdout","time":"2022-11-27T01:02:02.579983536Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:02:02.580Z\", \"log.level\": \"WARN\", \"message\":\"failed to retrieve shard stats from node [u0DuBr2mTH-d3fxDJbbhXg]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1079]\",\"log.logger\":\"org.elasticsearch.cluster.InternalClusterInfoService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\",\"error.type\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException\",\"error.message\":\"[es01][10.0.30.12:9300][indices:monitor/stats[n]] request_id [239967013] timed out after [14811ms]\",\"error.stack_trace\":\"org.elasticsearch.transport.ReceiveTimeoutTransportException: [es01][10.0.30.12:9300][indices:monitor/stats[n]] request_id [239967013] timed out after [14811ms]\\n\"}\n","stream":"stdout","time":"2022-11-27T01:02:05.425780842Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:02:02.768Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [15s/15011ms] ago, timed out [200ms/200ms] ago, action [indices:monitor/stats[n]], node [{es01}{u0DuBr2mTH-d3fxDJbbhXg}{h8GVRQ0ATf6UtONWcFYVWw}{es01}{10.0.30.12}{10.0.30.12:9300}{dhilmstw}{ml.machine_memory=6442450944, rack=r0, ml.max_jvm_size=3221225472, xpack.installed=true}], id [239967013]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#1]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:02:05.425808343Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:02:02.769Z\", \"log.level\": \"WARN\", \"message\":\"Received response for a request that has timed out, sent [15s/15011ms] ago, timed out [200ms/200ms] ago, action [cluster:monitor/nodes/stats[n]], node [{es01}{u0DuBr2mTH-d3fxDJbbhXg}{h8GVRQ0ATf6UtONWcFYVWw}{es01}{10.0.30.12}{10.0.30.12:9300}{dhilmstw}{ml.machine_memory=6442450944, rack=r0, ml.max_jvm_size=3221225472, xpack.installed=true}], id [239967008]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][transport_worker][T#1]\",\"log.logger\":\"org.elasticsearch.transport.TransportService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:02:05.425835443Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:02:03.951Z\", \"log.level\": \"WARN\", \"message\":\"health check of [/usr/share/elasticsearch/data] took [7606ms] which is above the warn threshold of [5s]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1132]\",\"log.logger\":\"org.elasticsearch.monitor.fs.FsHealthService\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:02:05.425841643Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:02:15.460Z\", \"log.level\": \"INFO\", \"message\":\"primary-replica resync completed with 0 operations\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1145]\",\"log.logger\":\"org.elasticsearch.index.shard.IndexShard\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\",\"tags\":[\" [.kibana-event-log-7.17.3-000002][0]\"]}\n","stream":"stdout","time":"2022-11-27T01:02:15.460284551Z"}
{"log":"{\"@timestamp\":\"2022-11-27T01:02:17.581Z\", \"log.level\":\"ERROR\", \"message\":\"collector [cluster_stats] timed out when collecting data: node [u0DuBr2mTH-d3fxDJbbhXg] did not respond within [10s]\", \"ecs.version\": \"1.2.0\",\"service.name\":\"ES_ECS\",\"event.dataset\":\"elasticsearch.server\",\"process.thread.name\":\"elasticsearch[es03][generic][T#1149]\",\"log.logger\":\"org.elasticsearch.xpack.monitoring.collector.cluster.ClusterStatsCollector\",\"elasticsearch.cluster.uuid\":\"o3uSGc7TR0O503-YKA9kLQ\",\"elasticsearch.node.id\":\"ElsMpBDUTa2m8LnuH-Boyg\",\"elasticsearch.node.name\":\"es03\",\"elasticsearch.cluster.name\":\"ds-monitoring-prod\"}\n","stream":"stdout","time":"2022-11-27T01:02:17.581805434Z"}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.