How can I stop cluster from filling my cold nodes with hot data?

Elastic Stack Elasticsearch
1

title really says it all.

I just added two cold nodes to my cluster -- these have slow disks and I don't want active indexes on them but I find that ES is treating them the same as the other nodes.

both nodes have node.attr.node_type: cold in their configuration

2

Are your hot nodes labeled as hot? Do your index templates apply the correct settings to force newly created indices to the hot nodes? Are you using ILM to manage index transitions and movement?

3

nodes are labeled as hot, warm or cold

some index use ILM others don't (yet).

I was waiting to get the cold nodes provisioned before bringing all the indexes into ILM (there are hundreds of daily indexes that need to be consolidated (by reindexing into an index that is under ILM) and moved to cold (frozen) storage.

I am trying to migrate an existing system that is several years on and set up before xpack was "free"

Oops I have just discovered that all searching is timing out on ES so I have stopped the cold nodes.

So should I make sure that all indexes are under ILM before enabling the cold nodes?

BTW can I roll directly from hot to cold by not enabling warm?

4

I have now (more or less) recovered from the disaster this caused and want to have another try at getting those nodes into the cluster.

How should I proceed to stop ES completely flooding these new node whose disks are slow and (at the moment) under provisioned.

5

You will need to provide a bit more information on your configuration and the current allocation please.

6

Thanks Mark!

Very happy to supply info but I am not quite sure what you want.

If I were to get all (or at least) most of the indexes under ILM -- which is the aim anyway, would that help.

There are currently a lot of small (< 50MB), old daily indexes that I plan to reindex into ILM and consolidate into chunks of order of 1GB.

I also use moloch -- now called Arkime (full packet capture) which has very large indexes with high write rates and some shards got moved onto the slow disks and killed the cluster. I thought I had these indexes under ILM and am now waiting on the "vendor" if he was talking about ES's ILM or something else that he was doing from the API. I strongly suspect the latter.

Reduced to its essentials my question is this: Can I stop particular indexes that are not under ILM from being allocated onto my warm and cold nodes?

Having just formulated the question and then searched I find shard-allocation-filtering which appears to do exactly what I need. Am I correct in thinking that putting node.roles: ["data_cold"] in my cold nodes (without "data_content") will do what I need?

And as I said, I am happy to give details, I just need to know which!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.