How to setup elasticsearch warm node

Elastic Stack Elasticsearch
1

I have 8 node elasticsearch cluster (3 Master+3Hot data node+2Warm data node).At beginning all the indices like metricbeat and system indices were on hot node and warm node after sometime i ran the following query.

PUT /_cluster/settings
{
  "persistent": {
   "cluster.routing.allocation.include.data":"hot",
    "cluster.routing.allocation.require.data":"hot",
    "indices.lifecycle.poll_interval":"1m"
  }
  , "transient": {
   "cluster.routing.allocation.include.data":"hot",
   "cluster.routing.allocation.require.data":"hot",
   "indices.lifecycle.poll_interval":"1m"
     }
}

all the indices move to the hot nodes.
my aim is to move the indices to the warm node after a certain period of time.. to implement this i am using one index called log-packetbeat-flow.

output {
 if[agent][type]=="packetbeat"
 {
if[type]=="flow"
   {
    elasticsearch {
    hosts => ["http://em1:9200","http://em2:9200"]
    manage_template => false
    ilm_rollover_alias => "log-pb-flow"
    ilm_pattern => "000001"
    ilm_policy => "packetbeat_flow"
  }
  }

Here is my ilm policy for packetbeat_flow_policy

{
  "packetbeat_flow_policy" : {
    "version" : 1,
    "modified_date" : "2021-02-17T09:17:30.480Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "warm"
              }
            },
            "set_priority" : {
              "priority" : 50
            }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "2m"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "0d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    }
  }
}

Index Template for packetbeat_flow

{
  "packetbeat_flow" : {
    "order" : 0,
    "index_patterns" : [
      "log-pb-flow-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "packetbeat_flow_policy",
          "rollover_alias" : "log-pb-flow"
        },
        "routing" : {
          "allocation" : {
            "require" : {
              "data" : "hot"
            }
          }
        },
        "refresh_interval" : "1s",
        "number_of_shards" : "1",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

Total Indices in a cluster

health status index                                uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .apm-agent-configuration             0JDboxYDTn2ShGM5Z1GLqg   1   1          0            0       418b           209b
green  open   .monitoring-logstash-7-mb-2021.02.19 aFlj3_RQThKfXLKd-ZrMTw   1   1      20328            0      3.5mb          1.7mb
green  open   .kibana_1                            mJ2jNlZoTRGgDeUvmiN5zw   1   1         99          118      8.6mb          4.3mb
green  open   .kibana-event-log-7.11.0-000001      J2B__HfLT02aErWjm-q2Hg   1   1          1            0     11.2kb          5.6kb
green  open   .apm-custom-link                     REBoTbXYTAK2kf-aPnu8tQ   1   1          0            0       416b           208b
green  open   log-pb-flow-2021.02.19-000922        nKUCYuRDRYO8tywWM1DmSg   1   1          0            0       416b           208b
green  open   .kibana_task_manager_1               3jBxyeppTXCtSPa9nUWn6g   1   1          8       106658     23.8mb         11.4mb
green  open   .monitoring-logstash-7-mb-2021.02.16 8cM5voJ_SwiWG4yMp-9oVw   1   1      12630            0      2.5mb          1.2mb
green  open   .monitoring-logstash-7-mb-2021.02.17 XOH1ZMxhSA2AmrXt2VaCjw   1   1      60763            0     10.6mb          5.3mb
green  open   .monitoring-logstash-7-mb-2021.02.18 AWM51V2SShmwGpAVxGdF5w   1   1      69120            0     10.5mb          5.2mb
green  open   .monitoring-es-7-mb-2021.02.16       B7UfFLVpQKCn4W1ooxsn9A   1   1      90105            0    137.2mb         68.6mb
green  open   .monitoring-kibana-7-mb-2021.02.19   jOgUsxmpTsWVoInmUEQZFQ   1   1       5324            0      2.2mb            1mb
green  open   metricbeat-7.11.0-2021.02.16-000001  _FhO7SGSRPqUfGm_5rd7Rw   1   1       3091            0        1mb        546.9kb
green  open   .monitoring-kibana-7-mb-2021.02.18   zxYeFV19RL-V_ZLNUrHWtQ   1   1      17280            0      6.7mb          3.3mb
green  open   .monitoring-es-7-mb-2021.02.17       D5UILdI9QvWYmlqjTP5NdA   1   1     253224            0    362.3mb        181.1mb
green  open   .monitoring-kibana-7-mb-2021.02.17   lf2INk15Sk28GqtyNf1CMg   1   1      17280            0        7mb          3.5mb
green  open   .monitoring-kibana-7-mb-2021.02.16   DfTjk6XCR-6QhTpjP12NOA   1   1       5094            0        2mb            1mb
green  open   .monitoring-es-7-mb-2021.02.18       4X-R2Bv8TomqblcJ4uXsRg   1   1     296146            0    403.2mb        204.2mb
green  open   .monitoring-es-7-mb-2021.02.19       wXdJPI3nSeGuADfQmt7CqA   1   1     101502            0    140.1mb         69.4mb

GET /_cat/shards/log-pb-flow-*?v&pretty

index                         shard prirep state   docs store ip            node
log-pb-flow-2021.02.19-000923 0     p      STARTED    0  208b xx.xx.xx.xx eh1
log-pb-flow-2021.02.19-000923 0     r      STARTED    0  208b xx.xx.xx.xx eh3

log-pb-flow index is rolling but not to warm node.

2

The problem is solved by following this doc

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.