Elasticsearch Cluster in Red State: Shard Allocation and Pending Tasks Issues

Elastic Stack Elasticsearch
1

I have installed Elasticsearch on Linux Ubuntu at 3 VMs, 1 VM is a dedicated master node, and rest 2 VMs have 5 data nodes(there are 2 data nodes on each). I have installed these 5 nodes with rpm.

The elasticsearch configuration at each node is:

node 1:

    cluster.name: my_cluster
	#cluster.initial_master_nodes: [data_node_1, data_node_2, data_node_3, data_node_4, master_node_1] 
	node.name: data_node_1
	node.attr.rack: d1
	node.roles: [data, master, voting_only]
	xpack.ml.enabled: false
	xpack.security.enabled: true
	xpack.security.transport.ssl.enabled: true
	xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
	xpack.security.transport.ssl.verification_mode: certificate
	xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
	xpack.security.transport.ssl.keystore.password: ""
	network.host: 170.11.1.11
	http.port: 7201
	transport.port: 6201
	discovery.seed_hosts: ["170.11.1.12:6201", "170.11.1.12:6202", "170.11.1.11:6201", "170.11.1.11:6202", "170.11.1.10:6201"]
	bootstrap.memory_lock: true

node 2:

    cluster.name: my_cluster
	#cluster.initial_master_nodes: [data_node_1, data_node_2, data_node_3, data_node_4, master_node_1] 
	node.name: data_node_2
	node.attr.rack: d2
	node.roles: [data, master, voting_only]
	xpack.ml.enabled: false
	xpack.security.enabled: true
	xpack.security.transport.ssl.enabled: true
	xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
	xpack.security.transport.ssl.verification_mode: certificate
	xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
	xpack.security.transport.ssl.keystore.password: ""
	network.host: 170.11.1.11
	http.port: 7202
	transport.port: 6202
	discovery.seed_hosts: ["170.11.1.12:6201", "170.11.1.12:6202", "170.11.1.11:6201", "170.11.1.11:6202", "170.11.1.10:6201"]
	bootstrap.memory_lock: true

node 3:

    cluster.name: my_cluster
	#cluster.initial_master_nodes: [data_node_1, data_node_2, data_node_3, data_node_4, master_node_1] 
	node.name: data_node_3
	node.attr.rack: d3
	node.roles: [data, master, voting_only]
	xpack.ml.enabled: false
	xpack.security.enabled: true
	xpack.security.transport.ssl.enabled: true
	xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
	xpack.security.transport.ssl.verification_mode: certificate
	xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
	xpack.security.transport.ssl.keystore.password: ""
	network.host: 170.11.1.12
	http.port: 7201
	transport.port: 6201
	discovery.seed_hosts: ["170.11.1.12:6201", "170.11.1.12:6202", "170.11.1.11:6201", "170.11.1.11:6202", "170.11.1.10:6201"]
	bootstrap.memory_lock: true

node 4:

    cluster.name: my_cluster
	#cluster.initial_master_nodes: [data_node_1, data_node_2, data_node_3, data_node_4, master_node_1] 
	node.name: data_node_4
	node.attr.rack: d4
	node.roles: [data, master, voting_only]
	xpack.ml.enabled: false
	xpack.security.enabled: true
	xpack.security.transport.ssl.enabled: true
	xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
	xpack.security.transport.ssl.verification_mode: certificate
	xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
	xpack.security.transport.ssl.keystore.password: ""
	network.host: 170.11.1.12
	http.port: 7202
	transport.port: 6202
	discovery.seed_hosts: ["170.11.1.12:6201", "170.11.1.12:6202", "170.11.1.11:6201", "170.11.1.11:6202", "170.11.1.10:6201"]
	bootstrap.memory_lock: true

node 5:

    cluster.name: my_cluster
	#cluster.initial_master_nodes: [data_node_1, data_node_2, data_node_3, data_node_4, master_node_1] 
	node.name: master_node_1
	node.attr.rack: m1
	node.roles: [master]
	xpack.ml.enabled: false
	xpack.security.enabled: true
	xpack.security.transport.ssl.enabled: true
	xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
	xpack.security.transport.ssl.verification_mode: certificate
	xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
	xpack.security.transport.ssl.keystore.password: ""
	network.host: 170.11.1.10
	http.port: 9200
	transport.port: 6201
	discovery.seed_hosts: ["170.11.1.12:6201", "170.11.1.12:6202", "170.11.1.11:6201", "170.11.1.11:6202", "170.11.1.10:6201"]
	bootstrap.memory_lock: true

All the nodes are running and have joined the cluster, but the cluster health is always red.

following is the APIs response:

<curl -u myElastic:abbcc 'http://170.11.1.10:9200/_cluster/state'

{
  "cluster_name": "my_cluster",
  "cluster_uuid": "RDMdaR3gRKOxJeQwf3SS0g",
  "version": 20,
  "state_uuid": "WT9J-_UmR16osuk4kJeKZw",
  "master_node": "QuwnetlvSUGi8Ro8qrEYOQ",
  "blocks": {
    "global": {
      "1": {
        "description": "state not recovered / initialized",
        "retryable": true,
        "disable_state_persistence": true,
        "levels": [
          "read",
          "write",
          "metadata_read",
          "metadata_write"
        ]
      }
    }
  },
  "nodes": {
    "8sKDqrYSS_24jxcZqZDiDQ": {
      "name": "data_node_1",
      "ephemeral_id": "6IB3kihNSEK9TGa3-9L95A",
      "transport_address": "170.11.1.11:6201",
      "external_id": "data_node_1",
      "attributes": {
        "transform.config_version": "10.0.0",
        "xpack.installed": "true",
        "rack": "d1"
      },
      "roles": [
        "data",
        "master",
        "voting_only"
      ],
      "version": "8.12.0",
      "min_index_version": 7000099,
      "max_index_version": 8500008
    },
    "fNcPoAaAR7q25nhEpnL0HQ": {
      "name": "data_node_2",
      "ephemeral_id": "Bood-e_STtKBky0ep-XTyA",
      "transport_address": "170.11.1.11:6202",
      "external_id": "data_node_2",
      "attributes": {
        "xpack.installed": "true",
        "transform.config_version": "10.0.0",
        "rack": "d2"
      },
      "roles": [
        "data",
        "master",
        "voting_only"
      ],
      "version": "8.12.0",
      "min_index_version": 7000099,
      "max_index_version": 8500008
    },
    "y5L9k2OqQIC_Vhewf3ZwzQ": {
      "name": "data_node_3",
      "ephemeral_id": "vkOIMz_3RO2RrptOOKZuYQ",
      "transport_address": "170.11.1.12:6201",
      "external_id": "data_node_3",
      "attributes": {
        "xpack.installed": "true",
        "transform.config_version": "10.0.0",
        "rack": "d3"
      },
      "roles": [
        "data",
        "master",
        "voting_only"
      ],
      "version": "8.12.0",
      "min_index_version": 7000099,
      "max_index_version": 8500008
    },
    "QuwnetlvSUGi8Ro8qrEYOQ": {
      "name": "master_node_1",
      "ephemeral_id": "Mf6XYD4VTTCWecdms52Ctg",
      "transport_address": "170.11.1.10:6201",
      "external_id": "master_node_1",
      "attributes": {
        "xpack.installed": "true",
        "transform.config_version": "10.0.0",
        "rack": "m1"
      },
      "roles": [
        "master"
      ],
      "version": "8.12.0",
      "min_index_version": 7000099,
      "max_index_version": 8500008
    },
    "Hthen6kDRxiI0g0JuG6G2A": {
      "name": "data_node_4",
      "ephemeral_id": "VXBHP0ueSn6G0ho9-hTrvw",
      "transport_address": "170.11.1.12:6202",
      "external_id": "data_node_4",
      "attributes": {
        "xpack.installed": "true",
        "transform.config_version": "10.0.0",
        "rack": "d4"
      },
      "roles": [
        "data",
        "master",
        "voting_only"
      ],
      "version": "8.12.0",
      "min_index_version": 7000099,
      "max_index_version": 8500008
    }
  },
  "nodes_versions": [
    {
      "node_id": "8sKDqrYSS_24jxcZqZDiDQ",
      "transport_version": "8560000",
      "mappings_versions": {
        ".triggered_watches": {
          "version": 1,
          "hash": -502826165
        },
        ".secrets-inference": {
          "version": 1,
          "hash": -1434574148
        },
        ".fleet-agents-7": {
          "version": 1,
          "hash": 1021268528
        },
        ".fleet-servers-7": {
          "version": 1,
          "hash": -916922632
        },
        ".ml-config": {
          "version": 1,
          "hash": 1455005892
        },
        ".fleet-policies-leader-7": {
          "version": 1,
          "hash": -1108172796
        },
        ".geoip_databases": {
          "version": 1,
          "hash": -305757839
        },
        ".security-tokens-7": {
          "version": 1,
          "hash": 576296021
        },
        ".snapshot-blob-cache": {
          "version": 1,
          "hash": 632712485
        },
        ".security-profile-8": {
          "version": 2,
          "hash": -909540896
        },
        ".search-app-1": {
          "version": 1,
          "hash": -501711141
        },
        ".watches": {
          "version": 1,
          "hash": -1045118511
        },
        ".fleet-artifacts-7": {
          "version": 1,
          "hash": -1593703898
        },
        ".query-rules-1": {
          "version": 1,
          "hash": 1647955624
        },
        ".transform-internal-007": {
          "version": 1,
          "hash": 1144737897
        },
        ".fleet-enrollment-api-keys-7": {
          "version": 1,
          "hash": -840564854
        },
        ".fleet-actions-7": {
          "version": 1,
          "hash": 961753072
        },
        ".tasks": {
          "version": 0,
          "hash": -945584329
        },
        ".ml-meta": {
          "version": 1,
          "hash": 976416370
        },
        ".fleet-secrets-7": {
          "version": 1,
          "hash": -745394230
        },
        ".security-7": {
          "version": 1,
          "hash": -1061511639
        },
        ".logstash": {
          "version": 1,
          "hash": -1058806351
        },
        ".ml-inference-000005": {
          "version": 3,
          "hash": 919553140
        },
        ".inference": {
          "version": 1,
          "hash": 645017346
        },
        ".async-search": {
          "version": 0,
          "hash": -1403744380
        },
        ".fleet-policies-7": {
          "version": 1,
          "hash": 762674907
        },
        ".synonyms-2": {
          "version": 1,
          "hash": -888080772
        }
      }
    },
    {
      "node_id": "fNcPoAaAR7q25nhEpnL0HQ",
      "transport_version": "8560000",
      "mappings_versions": {
        ".triggered_watches": {
          "version": 1,
          "hash": -502826165
        },
        ".secrets-inference": {
          "version": 1,
          "hash": -1434574148
        },
        ".fleet-agents-7": {
          "version": 1,
          "hash": 1021268528
        },
        ".fleet-servers-7": {
          "version": 1,
          "hash": -916922632
        },
        ".ml-config": {
          "version": 1,
          "hash": 1455005892
        },
        ".fleet-policies-leader-7": {
          "version": 1,
          "hash": -1108172796
        },
        ".geoip_databases": {
          "version": 1,
          "hash": -305757839
        },
        ".security-tokens-7": {
          "version": 1,
          "hash": 576296021
        },
        ".snapshot-blob-cache": {
          "version": 1,
          "hash": 632712485
        },
        ".security-profile-8": {
          "version": 2,
          "hash": -909540896
        },
        ".search-app-1": {
          "version": 1,
          "hash": -501711141
        },
        ".watches": {
          "version": 1,
          "hash": -1045118511
        },
        ".fleet-artifacts-7": {
          "version": 1,
          "hash": -1593703898
        },
        ".query-rules-1": {
          "version": 1,
          "hash": 1647955624
        },
        ".transform-internal-007": {
          "version": 1,
          "hash": 1144737897
        },
        ".fleet-enrollment-api-keys-7": {
          "version": 1,
          "hash": -840564854
        },
        ".fleet-actions-7": {
          "version": 1,
          "hash": 961753072
        },
        ".tasks": {
          "version": 0,
          "hash": -945584329
        },
        ".ml-meta": {
          "version": 1,
          "hash": 976416370
        },
        ".fleet-secrets-7": {
          "version": 1,
          "hash": -745394230
        },
        ".security-7": {
          "version": 1,
          "hash": -1061511639
        },
        ".logstash": {
          "version": 1,
          "hash": -1058806351
        },
        ".ml-inference-000005": {
          "version": 3,
          "hash": 919553140
        },
        ".async-search": {
          "version": 0,
          "hash": -1403744380
        },
        ".inference": {
          "version": 1,
          "hash": 645017346
        },
        ".fleet-policies-7": {
          "version": 1,
          "hash": 762674907
        },
        ".synonyms-2": {
          "version": 1,
          "hash": -888080772
        }
      }
    },
    {
      "node_id": "y5L9k2OqQIC_Vhewf3ZwzQ",
      "transport_version": "8560000",
      "mappings_versions": {
        ".secrets-inference": {
          "version": 1,
          "hash": -1434574148
        },
        ".triggered_watches": {
          "version": 1,
          "hash": -502826165
        },
        ".fleet-agents-7": {
          "version": 1,
          "hash": 1021268528
        },
        ".fleet-servers-7": {
          "version": 1,
          "hash": -916922632
        },
        ".fleet-policies-leader-7": {
          "version": 1,
          "hash": -1108172796
        },
        ".ml-config": {
          "version": 1,
          "hash": 1455005892
        },
        ".geoip_databases": {
          "version": 1,
          "hash": -305757839
        },
        ".security-tokens-7": {
          "version": 1,
          "hash": 576296021
        },
        ".snapshot-blob-cache": {
          "version": 1,
          "hash": 632712485
        },
        ".security-profile-8": {
          "version": 2,
          "hash": -909540896
        },
        ".search-app-1": {
          "version": 1,
          "hash": -501711141
        },
        ".watches": {
          "version": 1,
          "hash": -1045118511
        },
        ".fleet-artifacts-7": {
          "version": 1,
          "hash": -1593703898
        },
        ".query-rules-1": {
          "version": 1,
          "hash": 1647955624
        },
        ".transform-internal-007": {
          "version": 1,
          "hash": 1144737897
        },
        ".fleet-enrollment-api-keys-7": {
          "version": 1,
          "hash": -840564854
        },
        ".fleet-actions-7": {
          "version": 1,
          "hash": 961753072
        },
        ".tasks": {
          "version": 0,
          "hash": -945584329
        },
        ".fleet-secrets-7": {
          "version": 1,
          "hash": -745394230
        },
        ".ml-meta": {
          "version": 1,
          "hash": 976416370
        },
        ".security-7": {
          "version": 1,
          "hash": -1061511639
        },
        ".logstash": {
          "version": 1,
          "hash": -1058806351
        },
        ".ml-inference-000005": {
          "version": 3,
          "hash": 919553140
        },
        ".inference": {
          "version": 1,
          "hash": 645017346
        },
        ".async-search": {
          "version": 0,
          "hash": -1403744380
        },
        ".fleet-policies-7": {
          "version": 1,
          "hash": 762674907
        },
        ".synonyms-2": {
          "version": 1,
          "hash": -888080772
        }
      }
    },
    {
      "node_id": "QuwnetlvSUGi8Ro8qrEYOQ",
      "transport_version": "8560000",
      "mappings_versions": {
        ".secrets-inference": {
          "version": 1,
          "hash": -1434574148
        },
        ".triggered_watches": {
          "version": 1,
          "hash": -502826165
        },
        ".fleet-agents-7": {
          "version": 1,
          "hash": 1021268528
        },
        ".fleet-servers-7": {
          "version": 1,
          "hash": -916922632
        },
        ".ml-config": {
          "version": 1,
          "hash": 1455005892
        },
        ".fleet-policies-leader-7": {
          "version": 1,
          "hash": -1108172796
        },
        ".security-tokens-7": {
          "version": 1,
          "hash": 576296021
        },
        ".geoip_databases": {
          "version": 1,
          "hash": -305757839
        },
        ".snapshot-blob-cache": {
          "version": 1,
          "hash": 632712485
        },
        ".security-profile-8": {
          "version": 2,
          "hash": -909540896
        },
        ".search-app-1": {
          "version": 1,
          "hash": -501711141
        },
        ".watches": {
          "version": 1,
          "hash": -1045118511
        },
        ".fleet-artifacts-7": {
          "version": 1,
          "hash": -1593703898
        },
        ".query-rules-1": {
          "version": 1,
          "hash": 1647955624
        },
        ".transform-internal-007": {
          "version": 1,
          "hash": 1144737897
        },
        ".fleet-enrollment-api-keys-7": {
          "version": 1,
          "hash": -840564854
        },
        ".fleet-actions-7": {
          "version": 1,
          "hash": 961753072
        },
        ".tasks": {
          "version": 0,
          "hash": -945584329
        },
        ".ml-meta": {
          "version": 1,
          "hash": 976416370
        },
        ".fleet-secrets-7": {
          "version": 1,
          "hash": -745394230
        },
        ".security-7": {
          "version": 1,
          "hash": -1061511639
        },
        ".logstash": {
          "version": 1,
          "hash": -1058806351
        },
        ".ml-inference-000005": {
          "version": 3,
          "hash": 919553140
        },
        ".async-search": {
          "version": 0,
          "hash": -1403744380
        },
        ".inference": {
          "version": 1,
          "hash": 645017346
        },
        ".fleet-policies-7": {
          "version": 1,
          "hash": 762674907
        },
        ".synonyms-2": {
          "version": 1,
          "hash": -888080772
        }
      }
    },
    {
      "node_id": "Hthen6kDRxiI0g0JuG6G2A",
      "transport_version": "8560000",
      "mappings_versions": {
        ".triggered_watches": {
          "version": 1,
          "hash": -502826165
        },
        ".secrets-inference": {
          "version": 1,
          "hash": -1434574148
        },
        ".fleet-agents-7": {
          "version": 1,
          "hash": 1021268528
        },
        ".fleet-servers-7": {
          "version": 1,
          "hash": -916922632
        },
        ".ml-config": {
          "version": 1,
          "hash": 1455005892
        },
        ".fleet-policies-leader-7": {
          "version": 1,
          "hash": -1108172796
        },
        ".geoip_databases": {
          "version": 1,
          "hash": -305757839
        },
        ".security-tokens-7": {
          "version": 1,
          "hash": 576296021
        },
        ".snapshot-blob-cache": {
          "version": 1,
          "hash": 632712485
        },
        ".security-profile-8": {
          "version": 2,
          "hash": -909540896
        },
        ".search-app-1": {
          "version": 1,
          "hash": -501711141
        },
        ".watches": {
          "version": 1,
          "hash": -1045118511
        },
        ".fleet-artifacts-7": {
          "version": 1,
          "hash": -1593703898
        },
        ".query-rules-1": {
          "version": 1,
          "hash": 1647955624
        },
        ".transform-internal-007": {
          "version": 1,
          "hash": 1144737897
        },
        ".fleet-enrollment-api-keys-7": {
          "version": 1,
          "hash": -840564854
        },
        ".fleet-actions-7": {
          "version": 1,
          "hash": 961753072
        },
        ".tasks": {
          "version": 0,
          "hash": -945584329
        },
        ".ml-meta": {
          "version": 1,
          "hash": 976416370
        },
        ".fleet-secrets-7": {
          "version": 1,
          "hash": -745394230
        },
        ".security-7": {
          "version": 1,
          "hash": -1061511639
        },
        ".logstash": {
          "version": 1,
          "hash": -1058806351
        },
        ".ml-inference-000005": {
          "version": 3,
          "hash": 919553140
        },
        ".async-search": {
          "version": 0,
          "hash": -1403744380
        },
        ".inference": {
          "version": 1,
          "hash": 645017346
        },
        ".fleet-policies-7": {
          "version": 1,
          "hash": 762674907
        },
        ".synonyms-2": {
          "version": 1,
          "hash": -888080772
        }
      }
    }
  ],
  "nodes_features": [
    {
      "node_id": "8sKDqrYSS_24jxcZqZDiDQ",
      "features": [
        "features_supported",
        "health.dsl.info",
        "usage.data_tiers.precalculate_stats"
      ]
    },
    {
      "node_id": "Hthen6kDRxiI0g0JuG6G2A",
      "features": [
        "features_supported",
        "health.dsl.info",
        "usage.data_tiers.precalculate_stats"
      ]
    },
    {
      "node_id": "QuwnetlvSUGi8Ro8qrEYOQ",
      "features": [
        "features_supported",
        "health.dsl.info",
        "usage.data_tiers.precalculate_stats"
      ]
    },
    {
      "node_id": "fNcPoAaAR7q25nhEpnL0HQ",
      "features": [
        "features_supported",
        "health.dsl.info",
        "usage.data_tiers.precalculate_stats"
      ]
    },
    {
      "node_id": "y5L9k2OqQIC_Vhewf3ZwzQ",
      "features": [
        "features_supported",
        "health.dsl.info",
        "usage.data_tiers.precalculate_stats"
      ]
    }
  ],
  "metadata": {
    "cluster_uuid": "RDMdaR3gRKOxJeQwf3SS0g",
    "cluster_uuid_committed": false,
    "cluster_coordination": {
      "term": 35,
      "last_committed_config": [
        "{bootstrap-placeholder}-data_node_1",
        "{bootstrap-placeholder}-data_node_2",
        "y5L9k2OqQIC_Vhewf3ZwzQ",
        "QuwnetlvSUGi8Ro8qrEYOQ",
        "Hthen6kDRxiI0g0JuG6G2A"
      ],
      "last_accepted_config": [
        "{bootstrap-placeholder}-data_node_1",
        "{bootstrap-placeholder}-data_node_2",
        "y5L9k2OqQIC_Vhewf3ZwzQ",
        "QuwnetlvSUGi8Ro8qrEYOQ",
        "Hthen6kDRxiI0g0JuG6G2A"
      ],
      "voting_config_exclusions": [
        
      ]
    },
    "templates": {
      
    },
    "indices": {
      
    },
    "index-graveyard": {
      "tombstones": [
        
      ]
    },
    "reserved_state": {
      
    }
  },
  "routing_table": {
    "indices": {
      
    }
  },
  "routing_nodes": {
    "unassigned": [
      
    ],
    "nodes": {
      "y5L9k2OqQIC_Vhewf3ZwzQ": [
        
      ],
      "fNcPoAaAR7q25nhEpnL0HQ": [
        
      ],
      "8sKDqrYSS_24jxcZqZDiDQ": [
        
      ],
      "Hthen6kDRxiI0g0JuG6G2A": [
        
      ]
    }
  }
}

curl -u myElastic:abbcc 'http://170.11.1.10:9200/_cat/nodes'

170.11.1.11 3 70 0 0.00 0.00 0.00 dmv - data_node_1
170.11.1.11 2 70 0 0.00 0.00 0.00 dmv - data_node_2
170.11.1.12 0 70 0 0.00 0.00 0.00 dmv - data_node_3
170.11.1.10 34 74 0 0.00 0.01 0.00 m * master_node_1
170.11.1.12 2 70 0 0.00 0.00 0.00 dmv - data_node_4

curl -u myElastic:abbcc 'http://170.11.1.10:9200/_cluster/health'

{
  "cluster_name": "my_cluster",
  "status": "red",
  "timed_out": false,
  "number_of_nodes": 5,
  "number_of_data_nodes": 4,
  "active_primary_shards": 0,
  "active_shards": 0,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 5,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 5357481,
  "active_shards_percent_as_number": "NaN"
}

curl -u myElastic:abbcc 'http://170.11.1.10:9200/_cluster/pending_tasks'

{"tasks":
[
  {
    "insert_order": 5,
    "priority": "URGENT",
    "source": "reconfigure",
    "executing": false,
    "time_in_queue_millis": 5401712,
    "time_in_queue": "1.5h"
  },
  {
    "insert_order": 6,
    "priority": "URGENT",
    "source": "update_tsdb_data_stream_end_times",
    "executing": false,
    "time_in_queue_millis": 5101831,
    "time_in_queue": "1.4h"
  },
  {
    "insert_order": 2,
    "priority": "HIGH",
    "source": "cluster_reroute(post-join reroute)",
    "executing": false,
    "time_in_queue_millis": 5402112,
    "time_in_queue": "1.5h"
  },
  {
    "insert_order": 3,
    "priority": "NORMAL",
    "source": "update snapshot after shards changed [false] or node configuration changed [true]",
    "executing": false,
    "time_in_queue_millis": 5401912,
    "time_in_queue": "1.5h"
  },
  {
    "insert_order": 4,
    "priority": "NORMAL",
    "source": "local-gateway-elected-state",
    "executing": false,
    "time_in_queue_millis": 5401712,
    "time_in_queue": "1.5h"
  }
]
}

curl -u myElastic:abbcc 'http://170.11.1.10:9200/_settings'

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"
      }
    ],
    "type": "master_not_discovered_exception",
    "reason": "org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
    "caused_by": {
      "type": "cluster_block_exception",
      "reason": "blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"
    }
  },
  "status": 503
}

What could be causing the red status in my Elasticsearch cluster?

How can I address and resolve the pending tasks that seem to be affecting my cluster?

2

Hi @Sanjeet_Kumar Welcome to the community.

Curious if you read about and this very important setting.

cluster.initial_master_nodes

Please remember to remove it after master is discovered in your cluster forms.

Oh I just noticed that you also have all those voting only nodes.
I am not sure why you configured it that way... If your intention is to only have one master node only ever do not need those other voting nodes just a single master node.

Of course, we would generally recommend having three master eligible nodes (not voting only) for a cluster like this. Anyways, that is more resilient.

So I think you need to fix both of those

3

Hmm I don't think it's that, there's a master elected (hence the local-gateway-elected-state task). But it's weird that none of the tasks are marked as executing: true. @Sanjeet_Kumar what version is this, and can you share a thread dump of the elected master?

1 Like
4

Sorry I missed that you already shared that: 8.12.0

I'd be interested to see complete logs from all nodes too.

5

I was experiencing issues when using openjdk-21. After downgrading to openjdk-17, the issues were fixed. Therefore, I believe there is some issue with openjdk-21 for Elasticsearch 8.12.0.

6

Ah yes, if you're not using the bundled JDK and you upgraded to JDK21.0.2 then you will be hit by JDK-8323659 which fits these symptoms. We very strongly recommend using the bundled JDK.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.