Hot-Warm-Cold Log Transistion

Elastic Stack Elasticsearch
1

Objective: Implement Hot-Warm-Cold Architecture
At first when ingested from beats, data should be only be ingested and spread across only on Hot nodes. After 30-days logs moved to warm and then after 60 days to cold
(ELK-Stack Version: 7.8.0)

On the hot nodes, I did add the config as

cluster.routing.allocation.awareness.attributes: index_state
node.attr.index_state: hot

and on warm nodes, I added as the config as:

cluster.routing.allocation.awareness.attributes: index_state
node.attr.index_state: warm

But while observing, I seriously noticed that all newly ingested logs-shards got shared/stored on warm node as-well. As soon as I introduced the wam node into the cluster all the shards were spread across which defeats the purpose.

Can anyone please guide/help me on this issue

2

What does your ILM policy look like?

3

I had the same exact problem. The fix for my was to set 'node.data: false' on the warm node, however I am still having problems to move the data to my warm node after X days.

4

That is not a fix, it just means your warm node will not store any indexed data.

5

Have you added the required index setting to your index templates?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.