Searchable snapshots - cold phase - not working as expected

INDEX TEMPLATE

PUT _template/cold-demo
{
    "index_patterns": ["cold-demo*"],
    "settings": {
        "index" : {
          "lifecycle" : {
            "name" : "cold_retention_prod",
            "rollover_alias" : "cold-demo"
          },
          "routing" : {
            "allocation" : {
              "require" : {
                "data" : "hot"
              }
            }
          }
        }
    },
      "mappings" : { },
      "aliases" : { }
}

ILM POLICY

PUT _ilm/policy/cold_retention_prod
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "rollover": {
            "max_size": "25gb",
            "max_age": "1m"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "1m",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "require": {
              "data": "warm"
            }
          }
        }
      },
      "cold": {
        "min_age": "3m",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "cold-tier",
            "force_merge_index": true
          },
          "allocate": {
            "require": {
              "data": "cold"
            },
            "number_of_replicas": 0
          }
        }
      },
      "delete": {
        "min_age": "15m",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Hi ,

Above is our index template and test ILM policy which has 3 phases (we are testing this in our dev environment to add new cold phase to existing hot warm architecture)
While doing this , there are no problem with hot and warm phases.
Shards are allocated correctly to respective nodes.

But when the index enters cold phase , first it performs searchable snapshot action - which converts regular index to searchable snapshot index (original index name cold-demo-000001 is converted to restored-cold-demo-000001 ) and the index setting of the converted index has below property due to which the cold index shard is assigned randomly to one of the hot or warm or cold nodes.

"index.routing.allocation.include._tier_preference": "data_cold,data_warm,data_hot"

how do i assign cold index shard only to cold nodes ? is there some setting i m missing in ILM policy or index templates ?

I also saw that the searchable_snapshot action requires data tiers. The action uses the index.routing.allocation.include._tier_preference setting to mount the index directly to the phase’s corresponding data tier. But how do i set this using ilm policy or is there anything else to acheive this ?

i do see that cold phase uses fully mounted index option and primary shard is recovered on to data tier nodes from repository .
how do i make sure the recovered primary shard is assigned only to cold node while using ILM policy (in our case recovered shard is randomly assigned to one of the hot , warm or cold)

Please don't create multiple topics on the same question :slight_smile: