Lab Environment: 3 Master and 2 Data nodes. Just sent some data to cluster without building custom templates or ILM policies or mappings.
I added 2 more Data nodes and made the first two Hot and the new 2 Warm nodes
promptly has several unallocated shards because:
"allocate_explanation" : "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
"explanation" : "index has a preference for tiers [data_content], but no nodes for any of those tiers are available in the cluster"
OK. First lesson: Don't change node roles randomly without planning.
But to recover (in case I actually run into this IRL) do I:
- Update the index template for those indexes to point to the Hot nodes as preferred?
- Create an ILM that rolls data off quickly along with #1
- Build 2 new nodes that are Data to allow the shards to be allocated and then create a migration plan to make sure all new data goes to Hot then rolls to Warm until the Data nodes are empty then shut those down?
- Nuke from orbit and rebuild? (it IS just a lab build)
- Something else?