# Hot-Warm Architecture with ILM: Cluster health turns yellow **URL:** \ **Category:** Elasticsearch\ **Tags:** ilm-index-lifecycle-management\ **Created:** [January 5, 2021, 4:59pm UTC](https://discuss.elastic.co/t/hot-warm-architecture-with-ilm-cluster-health-turns-yellow/260244 "2021-01-05T16:59:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Eduardo\_Iglesias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eduardo_iglesias/32/78395_2.png) [@Eduardo\_Iglesias](https://discuss.elastic.co/u/Eduardo_Iglesias)\ **Post date:** [January 5, 2021, 4:59pm UTC](https://discuss.elastic.co/t/hot-warm-architecture-with-ilm-cluster-health-turns-yellow/260244/1 "2021-01-05T16:59:58Z")
Hello community, I'm trying to implement a hot warm architecture but I can't make it work, the cluster health turns yellow. Below are the details of my cluster and some outputs: ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/8/48b04e4cb4f755d02d0588ae8058b759994db44e.png) `GET _cat/indices` ```auto yellow open test-000001 DAtf63hjT8Gds8eD-7jnNw 1 1 0 0 283b 283b green open .kibana_task_manager DconFDotSeewnvb4x432gQ 1 1 2 0 25.6kb 12.8kb green open .security-7 Ae41LqCHRxO1X2QK3IFUkg 1 1 44 0 191.6kb 95.8kb green open index-a gzntOy4zT0SDD2OdJUIP8A 1 1 0 0 460b 230b green open index-b WP0J-9gUR6-h2zdFOQjKCg 1 1 0 0 460b 230b green open index-c FpfyNbpLRqq8ohTAfQ_JHw 1 1 0 0 460b 230b green open .kibana_1 qbl9kLmIS2SqzAO-yWelDw 1 1 12 3 70.6kb 35.3kb ``` `GET _cluster/health` ```auto { "cluster_name" : "my-test-cluster", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 7, "active_shards" : 13, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 92.85714285714286 } ``` The steps that I've followed are the following: 1 - Set the nodes attributes on elasticsearch.yml config file. ```auto node.attr.hwc: hot node.attr.hwc: warm ``` 2- Define the lifecycle policy through Kibana and its hot/warm phases. 3 - Define an index template that uses the policy. ```auto PUT _template/my_template { "index_patterns": ["test-*"], "settings": { "index.lifecycle.name": "my_hotwarm_policy", "index.lifecycle.rollover_alias": "test-alias", "index.routing.allocation.require.hwc": "hot" } } ``` 4 - Create an initial managed index. ```auto PUT test-000001 { "aliases": { "test-alias":{ "is_write_index": true } } } ``` After this steps the health of my cluster goes from green to yellow. Below is the result of GET \_cluster/allocation/explain. Even thought the below explanations messages are pretty much clear I'm not able to find the reason why is this happening, are there any additional steps to carry out or any additional configuration that needs to be added in my nodes? ```auto { "index" : "test-000001", "shard" : 0, "primary" : false, "current_state" : "unassigned", "unassigned_info" : { "reason" : "INDEX_CREATED", "at" : "2021-01-05T16:20:29.780Z", "last_allocation_status" : "no_attempt" }, "can_allocate" : "no", "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions" : [ { "node_id" : "dP5VjTcoThq0KBKPVgzu9A", "node_name" : "MasterNode", "transport_address" : "192.168.1.111:9300", "node_attributes" : { "ml.machine_memory" : "1880035328", "xpack.installed" : "true", "ml.max_open_jobs" : "20" }, "node_decision" : "no", "weight_ranking" : 1, "deciders" : [ { "decider" : "filter", "decision" : "NO", "explanation" : """node does not match index setting [index.routing.allocation.require] filters [hwc:"hot"]""" } ] }, { "node_id" : "0-FheIdDSMSUimXGFV-D3w", "node_name" : "DataNodeB", "transport_address" : "192.168.1.113:9300", "node_attributes" : { "ml.machine_memory" : "1880035328", "ml.max_open_jobs" : "20", "xpack.installed" : "true", "hwc" : "warm" }, "node_decision" : "no", "weight_ranking" : 2, "deciders" : [ { "decider" : "filter", "decision" : "NO", "explanation" : """node does not match index setting [index.routing.allocation.require] filters [hwc:"hot"]""" } ] }, { "node_id" : "J6ZvaGR5S2uV16f6Dxm2Og", "node_name" : "DataNode", "transport_address" : "192.168.1.112:9300", "node_attributes" : { "ml.machine_memory" : "1905201152", "ml.max_open_jobs" : "20", "xpack.installed" : "true", "hwc" : "hot" }, "node_decision" : "no", "weight_ranking" : 3, "deciders" : [ { "decider" : "same_shard", "decision" : "NO", "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[test-000001][0], node[J6ZvaGR5S2uV16f6Dxm2Og], [P], s[STARTED], a[id=si5u9H73S1W2YlYcFtibUg]]" } ] } ] } ``` Thanks in advance! Eduardo Iglesias ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 5, 2021, 5:13pm UTC](https://discuss.elastic.co/t/hot-warm-architecture-with-ilm-cluster-health-turns-yellow/260244/2 "2021-01-05T17:13:29Z")
it appears to me you have not set the number of replicas the default is 1 which means for every shard you will have 1 Primary and 1 Replica which will always reside on different nodes. The message above says it has no where to put the replica shards for either the hot or warm You have 2 options set replicas to 0 and you will be subject to data loss if a node dies or create 2 hot and 2 warm and have a replica. > **[Index modules | Elasticsearch Reference \[7.10\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings)** ---
**Author:** ![Eduardo\_Iglesias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eduardo_iglesias/32/78395_2.png) [@Eduardo\_Iglesias](https://discuss.elastic.co/u/Eduardo_Iglesias)\ **Post date:** [January 5, 2021, 5:36pm UTC](https://discuss.elastic.co/t/hot-warm-architecture-with-ilm-cluster-health-turns-yellow/260244/3 "2021-01-05T17:36:08Z")
Thanks for your prompt reply @stephenb, I'm getting the below when trying to set the shards to 0. I guess I'll need to try the other approach. ```auto { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Failed to parse value [0] for setting [index.number_of_shards] must be >= 1" } ], "type": "illegal_argument_exception", "reason": "Failed to parse value [0] for setting [index.number_of_shards] must be >= 1" }, "status": 400 } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 5, 2021, 5:38pm UTC](https://discuss.elastic.co/t/hot-warm-architecture-with-ilm-cluster-health-turns-yellow/260244/4 "2021-01-05T17:38:27Z")
You should set `index.number_of_replicas` to `0`, not `index.number_of_shards`. ---
**Author:** ![Eduardo\_Iglesias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eduardo_iglesias/32/78395_2.png) [@Eduardo\_Iglesias](https://discuss.elastic.co/u/Eduardo_Iglesias)\ **Post date:** [January 5, 2021, 5:43pm UTC](https://discuss.elastic.co/t/hot-warm-architecture-with-ilm-cluster-health-turns-yellow/260244/5 "2021-01-05T17:43:15Z")
@Christian_Dahlqvist @stephenb My mistake on the previous reply. ```auto PUT test-000001 { "aliases": { "test-alias":{ "is_write_index": true } }, "settings": { "number_of_shards": 1, "number_of_replicas": 0 } } ``` Cluster is now green ```auto green open test-000001 210iLgz0SjaK7UG8AATlUw 1 0 0 0 230b 230b green open .security-7 Ae41LqCHRxO1X2QK3IFUkg 1 1 44 0 191.6kb 95.8kb green open .kibana_task_manager DconFDotSeewnvb4x432gQ 1 1 2 0 25.6kb 12.8kb green open index-a gzntOy4zT0SDD2OdJUIP8A 1 1 1 0 7kb 3.5kb green open index-b WP0J-9gUR6-h2zdFOQjKCg 1 1 1 0 7kb 3.5kb green open index-c FpfyNbpLRqq8ohTAfQ_JHw 1 1 1 0 7kb 3.5kb green open .kibana_1 qbl9kLmIS2SqzAO-yWelDw 1 1 12 3 70.6kb 35.3kb ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 5, 2021, 5:46pm UTC](https://discuss.elastic.co/t/hot-warm-architecture-with-ilm-cluster-health-turns-yellow/260244/6 "2021-01-05T17:46:50Z")
Excellent, just keep in mind that you have no redundancy and if a node dies you are subject to data loss. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 5, 2021, 5:47pm UTC](https://discuss.elastic.co/t/hot-warm-architecture-with-ilm-cluster-health-turns-yellow/260244/7 "2021-01-05T17:47:37Z")
You may also want to set this in a index template so the correct settings apply also to new indices. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 8:20am UTC](https://discuss.elastic.co/t/hot-warm-architecture-with-ilm-cluster-health-turns-yellow/260244/8 "2022-11-04T08:20:58Z")