Node cold but phase hot

Elastic Stack Elasticsearch
1

Hello

I am running Version: 6.8

when I look at _cat/shards/$index it says that the node is cold.

image
image784×108 38.7 KB

When I look at $index/_ilm/explain it says the index is in phase hot:

{
  "indices": {
    "7a900401ea8845819a5cf59310c8ebb5-2020-h1": {
      "index": "7a900401ea8845819a5cf59310c8ebb5-2020-h1",
      "managed": true,
      "policy": "test",
      "lifecycle_date_millis": 1604960637706,
      "phase": "hot",
      "phase_time_millis": 1604960637792,
      "action": "complete",
      "action_time_millis": 1604960637853,
      "step": "complete",
      "step_time_millis": 1604960637853,
      "phase_execution": {
        "policy": "test",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 2,
        "modified_date_in_millis": 1604914547699
      }
    }
  }
}

This is my ILM policy:

{
  "test": {
    "version": 2,
    "modified_date": "2020-11-09T09:35:47.699Z",
    "policy": {
      "phases": {
        "cold": {
          "min_age": "30d",
          "actions": {
            "allocate": {
              "number_of_replicas": 0,
              "include": {},
              "exclude": {},
              "require": {
                "data": "cold"
              }
            },
            "set_priority": {
              "priority": 0
            }
          }
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "set_priority": {
              "priority": 100
            }
          }
        }
      }
    }
  }
}

Any ideas?
Does the document (time series data) affect the ILM policy? Because the timestamp of the documents are older than 30 days, but the index has been created a few hours ago.

2

How do you ensure that your index gets allocated on a hot node? The common approach there would be a setting in the underlying index template ("routing.allocation.include.temp": "hot" where temp is something you pick and need to set on the node as well with node.attr.temp=hot).

And I don't see any include or exclude rules for the shard allocation in the cold phase.

PS: We will formalize the tiers in the future a bit more — right now this is held together by some configs you set.

3

How can I fix the hot policy to add the correct allocation part?

4

It's an index setting: https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html#index-allocation-filters

If you haven't applied it at creation time through a template, you can just add it later on with the _settings endpoint on the index.

5 
node.attr.temp: hot
node.attr.data: hot

This is part of the node configuration. So I only need to adapt the template. Are the include and exclude rules for the cold phase mandatory? How do they look like? Do I even have to use the hot entry in the ILM or would only the cold phase be sufficient?

6
  1. You'll only need one of the node.attr (but you can of course also use both).
  2. In the hot phase you would normally configure a rollover (unless you use something like a daily or weekly index pattern).
  3. The phase after hot is warm. You could use cold instead but it is rather unusual to have cold while skipping warm.
  4. The warm phase could look something like this:
    "warm" : {
        "min_age": "30d",
        "actions" : {
            "readonly" : { },
            "allocate" : {
                "include" : {
                    "temp" : "warm"
7

We don't intend to use warm because we only want to differentiate between SSD and HDD storage. The index is using a half-yearly pattern.

Is it possible to use it like this or am I missing something? The goal is to have the index switched from hot node to cold node after 183d.

In the template: "routing.allocation.include.data": "hot"

ILM policy like this:

{
  "six_months": {
    "version": 3,
    "modified_date": "2020-11-09T09:35:47.699Z",
    "policy": {
      "phases": {
        "cold": {
          "min_age": "183d",
          "actions": {
            "allocate": {
              "number_of_replicas": 0,
              "require": {
                "data": "cold"
              }
            },
            "set_priority": {
              "priority": 0
            }
          }
        }
      }
    }
  }
}
8

I think you'll need to try it out (with a min_age of 1h or so). I'd expect this to work though I would still go for hot instead to make the index read-only and do a force-merge (those aren't available in cold from the top of my head).

Warm and cold are just naming conventions, but we don't have any explicit steps built-in that would force one on SSDs or the other on spinning disks.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.