Hot-Warm migration - Need to update PRIOR Template (Hot-Only) to recognize WARM Allocators are not valid targets

[Randy-312 (Randy Bovay)]

In moving to Hot-Warm nodes, our Prior Template did NOT have an instance configuration that used the box_type values. So, migrating a cluster to hot-warm configuration went fine, and it started proplerly using WARM node allocators.
However, the clusters using the OLD Template eventually started putting nodes (which were hot) onto the warm node allocators. Performance was not bad (on AWS), but we realize that this is problematic and are looking at how to fix this now.

Yes, our Admin-Console-Elasticsearch is Now ALSO on WARM.. which is clearly not ideal.

Should we
A) Modify the Template to use a new instance configuration id (HOT node format).
... If we do.. how can we roll this out? It isn't taking effect immediatly, so appears we need to run the plan for it to take effect.

B) Should we create a NEW Plan, and then change the Cluster to use the New Plan.
.... I know we couldn't do a hot/warm migration in one version, but not sure about 2.2.2
... Will this change the ElasticSearch cluster id if we do this?

C) something else

[Alex_Piggott]

Just to make sure I understand the sequence of events, is it approximately:

• You had a "all hot" ECE, with some clusters built on that (which had data.default as their instance config id)
• You added warm allocators, and were able to build some clusters that were hot:warm
• Over time the old clusters had instances moved and some of those moved onto warm nodes

There's a couple of options:

• You can edit the data.default instance config (I think from the UI) and add an allocator filter that would allow only hot nodes, then just move the instances that are incorrectly on warm nodes
  • (we may have UI and/or API restrictions on the default instance configs .. I don't have a running platform to hand to check .. but it should be immediately obvious from the UI if that's the case)
• Build a new instance config and replace the default instance config ids with the created ones (which will automatically migrate the instances back to the hot nodes)

I'd probably try the first one first

Alex

[Randy-312 (Randy Bovay)]

We have edited the data.default instance config for the default template. 2.2.2 does not restrict us from editing system templates apparently.

Then, as we perform our upgrade, ECE is placing the newly built nodes ONLY on the allocators tagged as hot. For our Hot-Warm, that's appropriately placing them too

We will do a MoveNode for the system clusters that are not upgradeable, but for discussion, any change that would trigger a node rebuild would also have the effect of a move.