Searchable snapshot action - in cold phase restores the index to random data tier instead of only cold data tier

Elastic Stack Elasticsearch
1

Searchable snapshot action - in cold phase restores the index to random data tier instead of only cold data tier as the searchable snapshot action ignores all the allocation filters and sets the default one as "index.routing.allocation.include._tier_preference": "data_cold,data_warm,data_hot"

INDEX TEMPLATE

PUT _template/cold-demo
{
    "index_patterns": ["cold-demo*"],
    "settings": {
        "index" : {
          "lifecycle" : {
            "name" : "cold_retention_prod",
            "rollover_alias" : "cold-demo"
          },
          "routing" : {
            "allocation" : {
              "require" : {
                "data" : "hot"
              }
            }
          }
        }
    },
      "mappings" : { },
      "aliases" : { }
}

ILM POLICY

PUT _ilm/policy/cold_retention_prod
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "rollover": {
            "max_size": "25gb",
            "max_age": "1m"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "1m",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "require": {
              "data": "warm"
            }
          }
        }
      },
      "cold": {
        "min_age": "3m",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "cold-tier",
            "force_merge_index": true
          },
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "delete": {
        "min_age": "15m",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Hi ,

Above is our index template and test ILM policy which has 3 phases (we are testing this in our dev environment to add new cold phase to existing hot warm architecture)
While doing this , there are no problem with hot and warm phases.
Shards are allocated correctly to respective nodes.

But when the index enters cold phase , first it performs searchable snapshot action - which converts regular index to searchable snapshot index (original index name cold-demo-000001 is converted to restored-cold-demo-000001 ) and the index setting of the converted index has below property due to which the cold index shard is assigned randomly to one of the hot or warm or cold nodes.

"index.routing.allocation.include._tier_preference": "data_cold,data_warm,data_hot"

how do i assign cold index shard only to cold nodes ? is there some setting i m missing in ILM policy or index templates ?

I also saw that the searchable_snapshot action requires data tiers. The action uses the index.routing.allocation.include._tier_preference setting to mount the index directly to the phase’s corresponding data tier. But how do i set this using ilm policy or is there anything else to acheive this ?

i do see that cold phase uses fully mounted index option and primary shard is recovered on to data tier nodes from repository .
how do i make sure the recovered primary shard is assigned only to cold node while using ILM policy (in our case recovered shard is randomly assigned to one of the hot , warm or cold)

2

This suggests you're not using data tiers, you're just setting node attributes. You should define your hot tier with node.roles: data_hot, similarly for warm and cold, then the _tier_preference filter will do the right thing.

1 Like
3

thank you , that helped ! i was under the assumption that attributes are enough

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.