Content_tier

[mutt13y (Stuart Gall)]

Hello, with reference to 7.10 and Data tiers | Elasticsearch Guide [master] | Elastic

Content tier nodes are usually optimized for query performance—​they prioritize processing power over IO throughput so they can process complex searches and aggregations and return results quickly. While they are also responsible for indexing, content data is generally not ingested at as high a rate as time series data such as logs and metrics. From a resiliency perspective the indices in this tier should be configured to use one or more replicas.

and

When you create an index, by default Elasticsearch sets index.routing.allocation.include._tier_preference to data_content to automatically allocate the index shards to the content tier.

So I do not have "content_tier" applied to my cold nodes
however ILM is failing to allocate shards to the cold nodes because all indecies have the tier_preference to content.

Not sure what is the right way to fix this. I want all new data to go to hot these have
node.roles: ["ingest", "data_hot", "data_content" ]

cold nodes have
node.roles["data_cold"]

The instructions seem to imply that cold nodes both should not have data_content and must have data_content.

Regards
Stuart

[warkolm (Mark Walkom)]

Can you share your policy and Elasticsearch settings?

[mutt13y (Stuart Gall)]

Not sure what you mean by policy but
one of the index in question is currently like this

{
  "chef-dev_cert-2020.11.19" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "7_Days"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            },
            "require" : {
              "class" : "cold"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "chef-dev_cert-2020.11.19",
        "creation_date" : "1605758083756",
        "unassigned" : {
          "node_left" : {
            "delayed_timeout" : "35m"
          }
        },
        "number_of_replicas" : "1",
        "uuid" : "fhB_yQ8NRxiViiKBANeEgg",
        "version" : {
          "created" : "7100099"
        }
      }
    }
  }
}

The initial template is this

{
  "index_templates" : [
    {
      "name" : "chef",
      "index_template" : {
        "index_patterns" : [
          "chef-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "lifecycle" : {
                "name" : "7_Days"
              },
              "routing" : {
                "allocation" : {
                  "require" : {
                    "class" : "hot"
                  }
                }
              }
            }
          }
        },
        "composed_of" : [ ],
        "priority" : 100
      }
    }
  ]
}

7_days ILM is like this

{
  "7_Days" : {
    "version" : 14,
    "modified_date" : "2020-11-13T21:24:18.048Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "2d",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 1,
              "include" : { },
              "exclude" : { },
              "require" : {
                "class" : "hot"
              }
            },
            "set_priority" : {
              "priority" : null
            }
          }
        },
        "cold" : {
          "min_age" : "5d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "class" : "cold"
              }
            },
            "set_priority" : {
              "priority" : null
            }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 50
            }
          }
        },
        "delete" : {
          "min_age" : "6d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    }
  }
}

The error is

  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "XXXXX",
      "node_name" : "XXXX",
      "transport_address" : "XXXXX",
      "node_attributes" : {
        "datacenter" : "XXXX",
        "xpack.installed" : "true",
        "class" : "cold",
        "transform.node" : "false"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "data_tier",
          "decision" : "NO",
          "explanation" : "index has a preference for tiers [data_content] and node does not meet the required [data_content] tier"
        }
      ]
    },

Stuart