Adding a warm node

Elastic Stack Elasticsearch
1

I've started with 4 ELK nodes:

Host 1 Logstash & Kibana
Host 2 Elasticsearch node-1
Host 3 Elasticsearch node-2
Host 4 Elasticsearch node-3

The stack is running 7.16.1 and has just over a years' worth of data. I've not added node.roles values for the Elasticsearch nodes and everything else is mostly default. Logstash is outputting to all three ES nodes.

So, what I want to do is move all data older than 8 months to an Elasticsearch node-4, which I've built and added to the cluster. I also added this to that new node's config:

node.roles: [ "data_warm" ]

I then used Kibana to turn on the warm stage as follows:

Screen Shot 2022-04-05 at 11.54.14 AM
Screen Shot 2022-04-05 at 11.54.14 AM1099×1366 158 KB

So, I do see some data getting added to the warm node, but looks like just two days worth, possibly. What I would like is to have all data older than 8 months to be moved off of the hot nodes and to the warm node.

Is there something more I need to do to get this? Also, do the hot nodes need node.roles specified, and if so, what would be the proper values? I see this in the docs, but it's not clear why I need them all:

node.roles: ["master", "ingest", "ml", "data_hot", "data_content"]

Thanks in advance for any help!

--C

2

Just so it's clear, you are using an ILM policy you've defined in Kibana, right?

3

Thanks! Correct, just as shown in the screenshot -- I've done nothing else.

--C

4

Did you attach the policy to all your indices?

5

Hmm, so when I updated that policy, it still showed that it's attached to all the indices. I guess that update isn't retroactive? I assume there's a an api call to attach it to all of them?

6

It's not, nope. Check out Manage existing indices | Elasticsearch Guide [8.11] | Elastic;

The simplest way to transition to managing your periodic indices with ILM is to configure an index template to apply a lifecycle policy to new indices. Once the index you are writing to is being managed by ILM, you can manually apply a policy to your older indices.

7

Oh, I need to reindex everything? Yikes, didn't expect that...OK, I'll look into that. Thanks!

8

No, you don't. You can manually apply the policy as per the last link in that - Configure a lifecycle policy | Elasticsearch Guide [8.1] | Elastic

9

Oh, OK, great. So basically, this?

curl -X POST "localhost:9200/ecs_logstash/_ilm/remove?pretty"

curl -X GET "localhost:9200/ecs_logstash?pretty"

curl -X POST "localhost:9200/ecs_logstash/_open?pretty"

curl -X PUT "localhost:9200/ecs_logstash/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index": {
    "lifecycle": {
      "name": "logstash-policy"
    }
  }
}
'
10

Should I use a wildcard or alias name in the curls? I.e., ecs_logstash-* or ecs_logstash

11

Here you have a single index? Alias?

What are the names of the other existing indices?

12

They're daily rollover indices from logstash. The alias is ecs_logstash, but the indices names are, for example:

ecs-logstash-2022.03.14-000822
ecs-logstash-2022.04.05-001903

13

Cool, thanks. Then you will want;

curl -X PUT "localhost:9200/ecs-logstash-2022*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index": {
    "lifecycle": {
      "name": "logstash-policy"
    }
  }
}
14

Awesome, thank you. I'll run this in the morning and update this thread.

Thanks again!

--C

15

You might want to drop the last 2 on the year, so it takes everything from 2020 onwards.

16

Got it! Will do. Thanks....