ILM : configuration and pricing

[icirco]

hello,

In my filebeat (7.4) configuration I setup an index template. In a recent Topic, I known that is important to setup setup.ilm.enabled to false to run an index template.

filebeat.yml :

 setup.ilm.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
setup.template.enabled: true
setup.template.name: "filebeat-mytest-%{[agent.version]}"
setup.template.pattern: "filebeat-mytest-%{[agent.version]}-*"

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  index: "filebeat-%{[agent.version]}-mytest-%{+yyyy.MM.dd}"

Now, I want to work with ILM policies on elasticsearch (7.4).
It means that if ilm is disable in filebeat configuration, it is not possible to use ILM policies ?

In my kibana interface, I have the menu to configure ILM policies. But when I read the documentation, it seems that ILM policies configuration is in the x-pack, so I need a licence for use it ?

[jakelandis]

ILM is falls under the Basic license, which is free to use. See Basic here.

You don't need to configure ILM through filebeat, you can manage it directly through your index template.

The important parts are index.lifecycle.name in the index template, and if you use rollover, you must create the first index yourself - described here: https://www.elastic.co/guide/en/elasticsearch/reference/current/_actions.html#ilm-rollover-action and then add theindex.lifecycle.rollover_alias to your index template. Those are the two main things that setup.ilm.enabled perform for you.

Manually managing ILM with agent.version in the index name can be challenging since the above steps need performed everytime you update the agent.version. I would advise to allow filebeat to manage the ILM setup if at all possible.

[icirco]

hi,

Thanks jakelandis, I'm now able to manage my index lifecycle. Here a sample :

PUT _template/mytest-network-test-tpl
{
  "index_patterns": ["mytest-network-test-*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "index.lifecycle.name": "policy_network", 
    "index.lifecycle.rollover_alias": "network-test"
  },
  "aliases": {
        "mytest-network-test" : {}
  }
}


PUT mytest-network-test-000001
{
  "aliases": {
    "network-test":{
      "is_write_index": true 
    }
  }
}

I still have a misunderstanding, it's about the filebeat definition when you want to modify the index name...

So, when I start my index lifecycle policy, what is the name of index when a index lifecycle policy is working ? Because of the iteration of the index name, filebeat configuration doesn't know the index name iterated ?

output.elasticsearch:
hosts: ["localhost:9200"]
index: "mytest-network-test-000001"
#index: "mytest-network-test-????"

[jakelandis]

(assuming you disabled ILM via beats and are managing only via a template)

You will need to tell Filebeat to write to through the write/rollover alias you defined. network-test . That way when the rollover condition is met, ILM does the rollover and it changes the alias to point from -000001 to -000002.

The mytest-network-test alias you have defined doesn't hurt anything, but is not needed for ILM.

[icirco]

hello jakelandis,

I'm lost...

filebeat.yml :

setup.ilm.enabled: false

setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 0
index.lifecycle.name: policy_network
index.lifecycle.rollover_alias: mytest-network-test

setup.template.enabled: true
setup.template.name: "mytest-network-test-tpl"
setup.template.pattern: "mytest-network-test-*"

output.elasticsearch:
index: "mytest-network-test"

I created my template in kibana like this :

PUT _template/mytest-network-test-tpl
{
"index_patterns": [
"mytest-network-test-*"
],
"settings": {
"index": {
"lifecycle": {
"name": "policy_network",
"rollover_alias": "mytest-network-test"
},
"number_of_shards": "1",
"number_of_replicas": "0"
}
},
"aliases": {
"mytest-network-test": {}
}
}

when I fill my index, first, it create the index called "mytest-network-test" but no rollover_alias is associate to the index....

[jakelandis]

Before any data is written you need to create the -000001 index with the alias. With the above config:

1. Stop beats from sending data.

DELETE mytest-network-test

(*note - the above will delete data)
3.

PUT mytest-network-test-000001
{
  "aliases": {
    "mytest-network-test":{
      "is_write_index": true 
    }
  }
}

4. Start up beats to start ingesting data.

This should be mostly unnecessary if you allow beats to manage ILM. The beats config allows for modification to the index settings , and you need custom templates you can use template ordering to override the beats defaults. I still advise to let beats manage ILM for you if at all possible.

[icirco]

Dear jakelandis,

Thanks a lot for all informations : everything is running well now !

My mistake was I didn't create the first index.

So, to summary all steps :

config filebeat.yml :

setup.ilm.enabled: false

setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 0
index.lifecycle.name: policy_network
index.lifecycle.rollover_alias: mytest-network-test

setup.template.enabled: true
setup.template.name: "mytest-network-test-tpl"
setup.template.pattern: "mytest-network-test-*"

output.elasticsearch:
index: "mytest-network-test"

create template index by specify rollover_alias :

PUT _template/mytest-network-test-tpl
{
"index_patterns": [
"mytest-network-test-*"
],
"settings": {
"index": {
"lifecycle": {
"name": "policy_network",
"rollover_alias": "mytest-network-test"
},
"number_of_shards": "1",
"number_of_replicas": "0"
}
}
}

create the FIRST index !

PUT mytest-network-test-000001
{
"aliases": {
"mytest-network-test":{
"is_write_index": true
}
}
}