Adding a warm node

Elastic Stack Elasticsearch
1

I've started with 4 ELK nodes:

Host 1 Logstash & Kibana
Host 2 Elasticsearch node-1
Host 3 Elasticsearch node-2
Host 4 Elasticsearch node-3

The stack is running 7.16.1 and has just over a years' worth of data. I've not added node.roles values for the Elasticsearch nodes and everything else is mostly default. Logstash is outputting to all three ES nodes.

So, what I want to do is move all data older than 8 months to an Elasticsearch node-4, which I've built and added to the cluster. I also added this to that new node's config:

node.roles: [ "data_warm" ]

I then used Kibana to turn on the warm stage as follows:

Screen Shot 2022-04-05 at 11.54.14 AM
Screen Shot 2022-04-05 at 11.54.14 AM1099×1366 158 KB

So, I do see some data getting added to the warm node, but looks like just two days worth, possibly. What I would like is to have all data older than 8 months to be moved off of the hot nodes and to the warm node.

Is there something more I need to do to get this? Also, do the hot nodes need node.roles specified, and if so, what would be the proper values? I see this in the docs, but it's not clear why I need them all:

node.roles: ["master", "ingest", "ml", "data_hot", "data_content"]

Thanks in advance for any help!

--C

2

Just so it's clear, you are using an ILM policy you've defined in Kibana, right?

3

Thanks! Correct, just as shown in the screenshot -- I've done nothing else.

--C

4

Did you attach the policy to all your indices?

5

Hmm, so when I updated that policy, it still showed that it's attached to all the indices. I guess that update isn't retroactive? I assume there's a an api call to attach it to all of them?

6

It's not, nope. Check out Manage existing indices | Elasticsearch Guide [8.11] | Elastic;

The simplest way to transition to managing your periodic indices with ILM is to configure an index template to apply a lifecycle policy to new indices. Once the index you are writing to is being managed by ILM, you can manually apply a policy to your older indices.

7

Oh, I need to reindex everything? Yikes, didn't expect that...OK, I'll look into that. Thanks!

8

No, you don't. You can manually apply the policy as per the last link in that - Configure a lifecycle policy | Elasticsearch Guide [8.1] | Elastic

9

Oh, OK, great. So basically, this?

curl -X POST "localhost:9200/ecs_logstash/_ilm/remove?pretty"

curl -X GET "localhost:9200/ecs_logstash?pretty"

curl -X POST "localhost:9200/ecs_logstash/_open?pretty"

curl -X PUT "localhost:9200/ecs_logstash/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index": {
    "lifecycle": {
      "name": "logstash-policy"
    }
  }
}
'
10

Should I use a wildcard or alias name in the curls? I.e., ecs_logstash-* or ecs_logstash

11

Here you have a single index? Alias?

What are the names of the other existing indices?

12

They're daily rollover indices from logstash. The alias is ecs_logstash, but the indices names are, for example:

ecs-logstash-2022.03.14-000822
ecs-logstash-2022.04.05-001903

13

Cool, thanks. Then you will want;

curl -X PUT "localhost:9200/ecs-logstash-2022*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index": {
    "lifecycle": {
      "name": "logstash-policy"
    }
  }
}
14

Awesome, thank you. I'll run this in the morning and update this thread.

Thanks again!

--C

15

You might want to drop the last 2 on the year, so it takes everything from 2020 onwards.

16

Got it! Will do. Thanks....

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.