Q: ILM and Beats index-template thoughts in a lab 7.10.x setup

[Andreas_Falk (Falk)]

Hi,

We are setting up a new 7.10.1 cluster and are struggling with some ILM and beat stuff

First up, when doing a simple template setup through Filebeat

docker run \
  docker.elastic.co/beats/filebeat:7.10.1 \
  setup -E setup.kibana.host=$my_ip:5601 \
  -E output.elasticsearch.hosts=["$my_ip:9200"]

And when we look in index_templates in kibana, it says "lagacy templates" with the default settings on filebeat?

We want things be as "default" as it can be for future compability.
We have been using elastic since the "good old ELK" 1.x times, and have jumped through some hoops because we done "non default configs" in the days
And now ILM has bitten us a few times when playing in test with index-templates, so we have decided to use KISS philosophy all out.

Q: Is it recommended to use the "new index template" when setting up beats for the first time?

setup.template.type: index
And then edit the Filebeat ILM policy after that after this document?

Tutorial: Manage Filebeat time-based indices | Elasticsearch Reference [7.10]...

In a docker lab setup we got it working with 1h rollovers to the next phase in hot/warm/cold.

There the warm/cold indexes is placed on the correct nodes.
But our hot indices it placed "randomly" on the hot, worm or cold nodes?

Q: How do we assign where the hot incices should be assigned?

Below are some info from the lab setup:

Here are a GET _cat/nodeattrs?h=name,attr,value

es12 ml.machine_memory 33676890112
es12 ml.max_open_jobs  20
es12 xpack.installed   true
es12 data              warm
es12 transform.node    true
es11 ml.machine_memory 33676890112
es11 ml.max_open_jobs  20
es11 xpack.installed   true
es11 data              hot
es11 transform.node    true
es03 ml.machine_memory 33676890112
es03 ml.max_open_jobs  20
es03 xpack.installed   true
es03 data              cold
es03 transform.node    true
es02 ml.machine_memory 33676890112
es02 ml.max_open_jobs  20
es02 xpack.installed   true
es02 data              warm
es02 transform.node    true
es01 ml.machine_memory 33676890112
es01 ml.max_open_jobs  20
es01 xpack.installed   true
es01 data              hot
es01 transform.node    true

And a GET _ilm/policy/filebeat

{
  "filebeat" : {
    "version" : 2,
    "modified_date" : "2020-12-20T18:54:34.744Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "0ms",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "warm"
              }
            }
          }
        },
        "cold" : {
          "min_age" : "1h",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 0,
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "cold"
              }
            },
            "freeze" : { }
          }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "1h"
            }
          }
        },
        "delete" : {
          "min_age" : "1d",
          "actions" : { }
        }
      }
    }
  }
}

--
Regards Falk