How to turn a data node into a dedicated ML node?

[Shubhangi]

Hi,
I've 8 data nodes right now in production. I want to test ML in production, how many dedicated nodes would ML require? Also, I won't be adding any new nodes, so, how can I make the existing data nodes, machine learning nodes?
How do I reroute shards allocated to that data node, because wouldn't ES allocate those shards back to balance the shards throughout the cluster?
My question is, how do I convert a data node into dedicated ML node after assigning the shards allocated to it to the rest of the cluster?

I'm using ES: 7.8
Kibana: 7.8.

Thanks!

[warkolm (Mark Walkom)]

https://www.elastic.co/guide/en/elasticsearch/reference/7.9/modules-node.html#node-roles describes how to do that.

[Shubhangi]

Thanks for the reply.
To safely distribute the shard data and decommission the node i'll have to use allocation filter instead of repurpose, right?

I understand if I have to make the existing node into a dedicated ML node then I'll have to use node.roles: ml option. And inorder to do that I'll have make changes in elasticsearch.yml, but since we have a common elasticsearch.yml file for all the nodes in the cluster how do I use that config option in command line?

Thanks!

[warkolm (Mark Walkom)]

You will want to use allocation filtering to move the shards off the node, before changing it's role.

And you need to change the role in the config file you mentioned.

[Shubhangi]

Mark, Any way to change the role while restarting the node in the command line?
Something like:
bin/elasticsearch -Dnode.roles=ml
And for data nodes:
bin/elasticsearch -Dnode.roles=data

[warkolm (Mark Walkom)]

Yes that should work, but you should really use the config file to make it explicit and permanent.

[Shubhangi]

Hi Mark,

I'm trying to enable ML for 1 month trail period. For that, I made this change in elasticsearch.yml:
xpack.ml.enabled: true.

Then I ran the following command on all the nodes:

nohup /ELK7/elasticsearch-7.8.0/bin/elasticsearch -Dnode.roles=[ data, master, ingest, remote_cluster_client, transform ] > /dev/null &

Sadly, it showed zero effect, no logs were logged, nothing.

I was going to use:

nohup /ELK7/elasticsearch-7.8.0/bin/elasticsearch -Dnode.roles=[ ml ] > /dev/null &

for dedicated ml node but since the master was not up, none of the nodes responded.

When I use:
nohup /ELK7/elasticsearch-7.8.0/bin/elasticsearch> /dev/null &

It works all right, but with the roles mentioned in command line, it refuses to even form a process(id).

Is there something wrong with command that I'm running? What am I doing wrong here?

Thanks.