Questions on Hot-Warm Deployment Runners and Data Migration

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

I have a deployment cluster created using the Default Deployment template and considering whether to shift the data to a hot-warm deployment because there is a need to shift data based on their age.

  1. Does the Hot-Warm cluster require dedicated runners ? Is a runner able to serve as a Hot node while also hosting deployment clusters created from the default deployment template?

  2. For the Hot-Warm deployment, is it possible to shift data from the hot node to the warm node on a per document basis rather than shifting the entire index itself?

  3. Are there any migration steps to shift data from a Default Deployment to a Hot-Warm deployment? Currently, the default deployment consists of 1 index with data ranging from the past 8 years till today. The plan is to have data from the most recent 3 years to be in the Hot node while anything older than that goes into the Warm node.

2

Does the Hot-Warm cluster require dedicated runners ? Is a runner able to serve as a Hot node while also hosting deployment clusters created from the default deployment template?

The problem is the opposite - once you split the allocators into hot and warm, you have to tell the default instance configs which ones to use, else a plan change on the default config deployments could move them onto the warm. I believe you can edit the filter on the default deployments (otherwise you have to create new ones and migrate the clusters across to them)

For the Hot-Warm deployment, is it possible to shift data from the hot node to the warm node on a per document basis rather than shifting the entire index itself?

I don't believe that is functionality that ILM offers: Tutorial: Automate rollover with ILM | Elasticsearch Guide [8.11] | Elastic

Are there any migration steps to shift data from a Default Deployment to a Hot-Warm deployment? Currently, the default deployment consists of 1 index with data ranging from the past 8 years till today. The plan is to have data from the most recent 3 years to be in the Hot node while anything older than that goes into the Warm node.

One extra migration step is alluded to above - you'd have to move the data into 8 yearly indices so that you could roll based on index pattern. Otherwise just setting up the ILM rules and moving the data I think

Alex

3

Thanks for the explanations. Regarding the point below,

The problem is the opposite - once you split the allocators into hot and warm, you have to tell the default instance configs which ones to use, else a plan change on the default config deployments could move them onto the warm. I believe you can edit the filter on the default deployments (otherwise you have to create new ones and migrate the clusters across to them)

Does this refer to tagging the allocators such that the default deployments will only use allocators in the hot node?

4

The problem is that default deployments start with no filters. So it doesn't matter what you tag the allocators with, until you have also added a filter statement to the instance configs used by default deployment templates.

1 Like
5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.