ILM Hot, Warm, Cold?

Elastic Stack Elasticsearch
#1

Hello,
When I try to setup ILM to make an index to become "Warm" after 30 days, then "Cold" after 60 days, and deleted after 90 days, I get the following:

You can't control shard allocation without node attributes.

Can someone help me with what to do next?

Thanks!

#2

To get started, have a look here and here.

#3

Hi @lamp123432

This warning is telling you that you need to configure your Elasticsearch nodes before using ILM in this way. Open your elasticsearch.yml and use the following setting:

node.attr.temp: warm

This will tell Elasticsearch where to "move" indices which have been declared as "warm" by your ILM. For more details refer to the links @egalpin posted

#4

Hello, thanks for the reply. If I only have one node, does this mean I cannot use this feature?

Even with multiple node, one node should be dedicated to "warm" indexes while other nodes to hot and cold?

#5

Hey
well if you have a single node setup, then a hot-warm-cold architecture doesn't make much sense. The background is cost saving. So if you store sequential, time-based data (for example logs), after a while the data gets less relevant and indexing/querying doesn't need to be as fast and performant any more.

Example:

  • 3 data nodes – one is "hot", one is "warm" and one is "cold"
  • The "hot" node is running on really expensive, super fast hardware, while "warm" and "cold" are running on cheap, old hardware
  • Now you can configure your Cluster so only the most recents logs (for example past 14 days) remain on the "hot" node --> super fast indexing and querying
  • After 14 days the indices are rolled over to "warm"
  • After 28 days your indices get rolled over to "cold"
#6

Great explanation, I do have a client that has 3 nodes, so this will be beneficial.

But I have a second client with only one node, after some time Logstash cannot send anything to Elasticsearch because it complains that all the shards have been allocated. It happens after 3 months, then I have to delete old indexes. So that's why I thought ILM policies could help keep the data longer.

In this case, do you think roll-ups would be my only option?

#7

Do you have a more precise error? All shards being "allocated" is actually a good thing and should not cause any problems.
Nevertheless, using Index Lifecycle Policies is best practice and saves you a lot of manual work and disc space when configured correctly :wink:
You can use those policies no matter how many nodes you have - just moving shards based on "temperature" won't be possible on a single node cluster

#8

Here's the error:

Dec 04 18:01:09 node1 logstash[1578]: [2020-12-04T18:01:09,812][WARN ][logstash.outputs.elasticsearch][netflow][4bs9s93560b30d657e23c4241487efc] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netflow-node1-2020.12.04", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x6c768f05>], :response=>{"index"=>{"_index"=>"netflow-node1-2020.12.04", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Validation Failed: 1: this action would add [6] total shards, but this cluster currently has [2000]/[2000] maximum shards open;"}}}}

Logstash cannot send new data to Elasticsearch, because a new index would be created and there is no enough shards available.

This seems to tell me that too many shards are open, I need to close a few of them. So without having dedicated "warm" or "cold" nodes, my only option is to delete indexes to get back some shards?

#10

What is the output of:

GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

If some outputs are too big, please share them on gist.github.com and link them here.

#11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.