Hot/Warm/Cold index assignment

Hi,

I've set up a 3 node cluster for a testing stack. Each node is set up on Docker with the required: node.attr.data=hot / node.attr.data=warm / node.attr.data=cold respectivly.
I have created a policy in Kibana as follows:

PUT _ilm/policy/cold_policy_3_MB
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "3mb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "number_of_replicas": 1,
            "include": {},
            "exclude": {},
            "require": {
              "data": "cold"
            }
          },
          "set_priority": {
            "priority": 0
          }
        }
      }
    }
  }
}

I have assigned this policy to an auditbeat index. What I was expecting to see here is the auditbeat index ending up initially in the HOT node and when it hits 3MB in size, it rolls over and creates another index on the HOT node (it's set up for 1 primary, no replicas). Once those rollovers are an hour old, they are moved to the COLD node. This isn't happening. The issues I'm seeing is that the index initially ends up on the WARM node and never seems to touch the HOT node. Also, the index is allowed to climb above the specified 3MB without rolling over.
I assumed that by default, Elasticsearch would push the new index to the HOT node by default. I also assumed that as I have assigned this policy to this particular index, those settings would be applied to the index. I can see that the ILM policy has been assigned to the index, I'm just not sure why the settings are not being adhered to.

Any suggestions would be appreciated.

Hi,

You'll need to assign the index.routing.allocation.require setting initially when the index is created in Your cluster. Here we're using a template to ensure that all indices are created on a hot node:

      "routing": {
        "allocation": {
          "require": {
            "temp": "hot"
          }
        }
      },

For debugging why the rollover is not working, I'd suggest using the Rollover API with the ?dry-run mode:

POST /<indexname>/_rollover?dry_run
{
  "conditions" : {
    "max_age": "1d",
    "max_size": "3mb"
  }
}

Regards,
Alex