Hello everyone,
I am working on a project with elasticsearch 9.4.1, where the goal is to move a specific data stream (ds) to a dedicated cold node for that data stream.
Here are the nodes I have prepared:
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
localhost-3 70 61 0 0.03 0.04 0.00 cm - node-3-cold
localhost-1 19 96 3 0.06 0.10 0.07 hms * node-1-hot
localhost-2 14 66 1 0.02 0.05 0.01 cm - node-2-cold
node host ip attr value
node-1-hot 192.168.71.192 192.168.71.192 transform.config_version 10.0.0
node-1-hot 192.168.71.192 192.168.71.192 xpack.installed true
node-1-hot 192.168.71.192 192.168.71.192 ml.config_version 12.0.0
node-2-cold 192.168.71.193 192.168.71.193 xpack.installed true
node-2-cold 192.168.71.193 192.168.71.193 dedicated_node itlea
node-2-cold 192.168.71.193 192.168.71.193 ml.config_version 12.0.0
node-2-cold 192.168.71.193 192.168.71.193 transform.config_version 10.0.0
node-3-cold 192.168.71.194 192.168.71.194 xpack.installed true
node-3-cold 192.168.71.194 192.168.71.194 transform.config_version 10.0.0
node-3-cold 192.168.71.194 192.168.71.194 ml.config_version 12.0.0
Sample indices/data streams I am testing:
-logs-itlea*
-template-dummy*
The plan is as follows:
logs-itlea* -> node-2-cold
template-dummy* -> node-3-cold
For the dummy data stream, it worked exactly as expected; it successfully moved to node-3 without anything going to node-2.
index shard prirep state docs store dataset ip node
.ds-template-dummy-2026.05.25-2026.05.25-000001 0 p STARTED 218 326.6kb 326.6kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000002 0 p STARTED 179 221.8kb 221.8kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000003 0 p STARTED 179 223kb 223kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000004 0 p STARTED 179 222.4kb 222.4kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000005 0 p STARTED 179 222.2kb 222.2kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000006 0 p STARTED 178 222kb 222kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000007 0 p STARTED 179 222.2kb 222.2kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000008 0 p STARTED 179 223kb 223kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000009 0 p STARTED 179 222.7kb 222.7kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000010 0 p STARTED 178 222.6kb 222.6kb 192.168.71.194 node-3-cold
.ds-template-dummy-2026.05.25-2026.05.25-000011 0 p STARTED 179 279.7kb 279.7kb 192.168.71.192 node-1-hot
.ds-template-dummy-2026.05.25-2026.05.25-000012 0 p STARTED 129 201.9kb 201.9kb 192.168.71.192 node-1-hot
However, for logs-itlea*, it is not working as intended. It hasn't moved to node-2 at all.
index shard prirep state docs store dataset ip node
.ds-logs-itlea-2026.05.25-2026.05.25-000001 0 p STARTED 179 143.7kb 143.7kb 192.168.71.192 node-1-hot
.ds-logs-itlea-2026.05.25-2026.05.25-000002 0 p STARTED 179 198.1kb 198.1kb 192.168.71.192 node-1-hot
.ds-logs-itlea-2026.05.25-2026.05.25-000003 0 p STARTED 178 143.3kb 143.3kb 192.168.71.192 node-1-hot
.ds-logs-itlea-2026.05.25-2026.05.25-000004 0 p STARTED 179 143kb 143kb 192.168.71.192 node-1-hot
.ds-logs-itlea-2026.05.25-2026.05.25-000005 0 p STARTED 179 143.6kb 143.6kb 192.168.71.192 node-1-hot
.ds-logs-itlea-2026.05.25-2026.05.25-000006 0 p STARTED 178 143.4kb 143.4kb 192.168.71.192 node-1-hot
.ds-logs-itlea-2026.05.25-2026.05.25-000007 0 p STARTED 179 199.4kb 199.4kb 192.168.71.192 node-1-hot
.ds-logs-itlea-2026.05.25-2026.05.25-000008 0 p STARTED 179 143.7kb 143.7kb 192.168.71.192 node-1-hot
The index settings look like this:
{
".ds-logs-itlea-2026.05.25-2026.05.25-000001": {
"settings": {
"index": {
"lifecycle": {
"name": "itlea_policy",
"indexing_complete": "true"
},
"mode": "standard",
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_hot"
},
"require": {
"dedicated_node": "itlea"
}
}
},
"hidden": "true",
"number_of_shards": "1",
"provided_name": ".ds-logs-itlea-2026.05.25-2026.05.25-000001",
"creation_date": "1779715174846",
"number_of_replicas": "0",
"uuid": "Eq5dR-vlTWiYJV038eIb1g",
"version": {
"created": "9094000"
}
}
}
}
}
The ILM explain for logs-itlea looks like this:
{
"indices": {
".ds-logs-itlea-2026.05.25-2026.05.25-000001": {
"index": ".ds-logs-itlea-2026.05.25-2026.05.25-000001",
"managed": true,
"policy": "itlea_policy",
"index_creation_date_millis": 1779715174846,
"time_since_index_creation": "48.91m",
"lifecycle_date_millis": 1779715350945,
"age": "45.98m",
"age_in_millis": 2758991,
"phase": "cold",
"phase_time_millis": 1779715531052,
"action": "allocate",
"action_time_millis": 1779715531052,
"step": "check-allocation",
"step_time_millis": 1779715531454,
"step_info": {
"message": "Waiting for [1] shards to be allocated to nodes matching the given filters",
"shards_left_to_allocate": 1,
"all_shards_active": true,
"number_of_replicas": 0
},
"phase_execution": {
"policy": "itlea_policy",
"phase_definition": {
"min_age": "3m",
"actions": {
"allocate": {
"include": {},
"exclude": {},
"require": {
"dedicated_node": "itlea"
}
}
}
},
"version": 3,
"modified_date_in_millis": 1779715080634
},
"skip": false
}
}
}
In the elasticsearch.yml, I have already added the node.attr for node-2.
cluster.name: my-cluster
node.name: node-2-cold
node.roles: [master, data_cold]
node.attr.dedicated_node: "itlea"
And in the ILM policy, I have also configured/enabled that node.attr.
itlea_policy
Hot phase
Rollover
Maximum primary shard size: 50gb
Maximum age: 2m
Cold phase
Move data into phase when
3m old
Data allocation
Node attributes: {"dedicated_node":"itlea"}
For refrence this is configuration on node3
cluster.name: my-cluster
node.name: node-3-cold
node.roles: [master, data_cold]
Previously, I tried running the following command, and the index successfully moved to node-2. However, when I try adding this to the index template, it doesn't move at all:
PUT /.ds-logs-itlea-2026.05.25-2026.05.25-*/_settings
{
"index.routing.allocation.include._tier_preference": null
}
{
"index": {
"lifecycle": {
"name": "itlea_policy"
},
"routing": {
"allocation": {
"include": {
"_tier_preference": null
}
}
},
"number_of_replicas": "0",
"mode": "standard"
}
}
There is no way I should run the command above manually every day or every time the index rolls over.
Has anyone encountered a similar problem? any insight into why the index isn't moving to node-2?
If you need any additional data, I can provide it.
Thank you.