The stack is running 7.16.1 and has just over a years' worth of data. I've not added node.roles values for the Elasticsearch nodes and everything else is mostly default. Logstash is outputting to all three ES nodes.
So, what I want to do is move all data older than 8 months to an Elasticsearch node-4, which I've built and added to the cluster. I also added this to that new node's config:
node.roles: [ "data_warm" ]
I then used Kibana to turn on the warm stage as follows:
So, I do see some data getting added to the warm node, but looks like just two days worth, possibly. What I would like is to have all data older than 8 months to be moved off of the hot nodes and to the warm node.
Is there something more I need to do to get this? Also, do the hot nodes need node.roles specified, and if so, what would be the proper values? I see this in the docs, but it's not clear why I need them all:
The simplest way to transition to managing your periodic indices with ILM is to configure an index template to apply a lifecycle policy to new indices. Once the index you are writing to is being managed by ILM, you can manually apply a policy to your older indices.