Hot-Warm Architecture with ILM: Cluster health turns yellow

Elastic Stack Elasticsearch
1

Hello community, I'm trying to implement a hot warm architecture but I can't make it work, the cluster health turns yellow. Below are the details of my cluster and some outputs:

image
GET _cat/indices

yellow open test-000001          DAtf63hjT8Gds8eD-7jnNw 1 1  0 0    283b   283b
green  open .kibana_task_manager DconFDotSeewnvb4x432gQ 1 1  2 0  25.6kb 12.8kb
green  open .security-7          Ae41LqCHRxO1X2QK3IFUkg 1 1 44 0 191.6kb 95.8kb
green  open index-a              gzntOy4zT0SDD2OdJUIP8A 1 1  0 0    460b   230b
green  open index-b              WP0J-9gUR6-h2zdFOQjKCg 1 1  0 0    460b   230b
green  open index-c              FpfyNbpLRqq8ohTAfQ_JHw 1 1  0 0    460b   230b
green  open .kibana_1            qbl9kLmIS2SqzAO-yWelDw 1 1 12 3  70.6kb 35.3kb

GET _cluster/health

{
  "cluster_name" : "my-test-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 7,
  "active_shards" : 13,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 92.85714285714286
}

The steps that I've followed are the following:

1 - Set the nodes attributes on elasticsearch.yml config file.

 node.attr.hwc: hot
 node.attr.hwc: warm

2- Define the lifecycle policy through Kibana and its hot/warm phases.
3 - Define an index template that uses the policy.

PUT _template/my_template
{
  "index_patterns": ["test-*"], 
  "settings": {
    "index.lifecycle.name": "my_hotwarm_policy", 
    "index.lifecycle.rollover_alias": "test-alias",
    "index.routing.allocation.require.hwc": "hot"
  }
}

4 - Create an initial managed index.

PUT test-000001 
{
  "aliases": {
    "test-alias":{
      "is_write_index": true 
    }
  }
}

After this steps the health of my cluster goes from green to yellow. Below is the result of GET _cluster/allocation/explain. Even thought the below explanations messages are pretty much clear I'm not able to find the reason why is this happening, are there any additional steps to carry out or any additional configuration that needs to be added in my nodes?

{
  "index" : "test-000001",
  "shard" : 0,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2021-01-05T16:20:29.780Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "dP5VjTcoThq0KBKPVgzu9A",
      "node_name" : "MasterNode",
      "transport_address" : "192.168.1.111:9300",
      "node_attributes" : {
        "ml.machine_memory" : "1880035328",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20"
      },
      "node_decision" : "no",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "filter",
          "decision" : "NO",
          "explanation" : """node does not match index setting [index.routing.allocation.require] filters [hwc:"hot"]"""
        }
      ]
    },
    {
      "node_id" : "0-FheIdDSMSUimXGFV-D3w",
      "node_name" : "DataNodeB",
      "transport_address" : "192.168.1.113:9300",
      "node_attributes" : {
        "ml.machine_memory" : "1880035328",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "hwc" : "warm"
      },
      "node_decision" : "no",
      "weight_ranking" : 2,
      "deciders" : [
        {
          "decider" : "filter",
          "decision" : "NO",
          "explanation" : """node does not match index setting [index.routing.allocation.require] filters [hwc:"hot"]"""
        }
      ]
    },
    {
      "node_id" : "J6ZvaGR5S2uV16f6Dxm2Og",
      "node_name" : "DataNode",
      "transport_address" : "192.168.1.112:9300",
      "node_attributes" : {
        "ml.machine_memory" : "1905201152",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "hwc" : "hot"
      },
      "node_decision" : "no",
      "weight_ranking" : 3,
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[test-000001][0], node[J6ZvaGR5S2uV16f6Dxm2Og], [P], s[STARTED], a[id=si5u9H73S1W2YlYcFtibUg]]"
        }
      ]
    }
  ]
}

Thanks in advance!
Eduardo Iglesias

2

it appears to me you have not set the number of replicas the default is 1 which means for every shard you will have 1 Primary and 1 Replica which will always reside on different nodes.

The message above says it has no where to put the replica shards for either the hot or warm

You have 2 options set replicas to 0 and you will be subject to data loss if a node dies or create 2 hot and 2 warm and have a replica.

1 Like
3

Thanks for your prompt reply @stephenb, I'm getting the below when trying to set the shards to 0. I guess I'll need to try the other approach.

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Failed to parse value [0] for setting [index.number_of_shards] must be >= 1"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Failed to parse value [0] for setting [index.number_of_shards] must be >= 1"
  },
  "status": 400
}
4

You should set index.number_of_replicas to 0, not index.number_of_shards.

1 Like
5

@Christian_Dahlqvist @stephenb My mistake on the previous reply.

PUT test-000001 
{
  "aliases": {
    "test-alias":{
      "is_write_index": true 
    }
  },
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  }
}

Cluster is now green

green open test-000001          210iLgz0SjaK7UG8AATlUw 1 0  0 0    230b   230b
green open .security-7          Ae41LqCHRxO1X2QK3IFUkg 1 1 44 0 191.6kb 95.8kb
green open .kibana_task_manager DconFDotSeewnvb4x432gQ 1 1  2 0  25.6kb 12.8kb
green open index-a              gzntOy4zT0SDD2OdJUIP8A 1 1  1 0     7kb  3.5kb
green open index-b              WP0J-9gUR6-h2zdFOQjKCg 1 1  1 0     7kb  3.5kb
green open index-c              FpfyNbpLRqq8ohTAfQ_JHw 1 1  1 0     7kb  3.5kb
green open .kibana_1            qbl9kLmIS2SqzAO-yWelDw 1 1 12 3  70.6kb 35.3kb
6

Excellent, just keep in mind that you have no redundancy and if a node dies you are subject to data loss.

2 Likes
7

You may also want to set this in a index template so the correct settings apply also to new indices.

1 Like