LifeCycle Policy Configuration

Elastic Stack Elasticsearch
1

Hi,

I have this simple scenario.
I have more or less 100 hosts (linux and windows) and i want to gather only logs and metrics. So i have install filebeat and metricbeat to all linux hosts and winlogbeat and metricbeat to all windows hosts.
From filebeat i am getting logs from syslog and from winlogbeat logs from Event Viewer. Also i want to save the logs to some output files with .log format that's why i have installed also logstash. So for the logs i am using also logstash (filebeat send to logstash and then to elasticsearch i suppose) and for the metrics only elasticsearch.
I have install ELK to 7 nodes. In 3 nodes i have elasticsearch, kibana and logstash(in these 3 nodes i am sending the logs from filebeat and winlogbeat using logstash output on their configuration files) and in the other 4 nodes i have only elasticsearch(so i am sending the metrcis from metricbeat to all 7 nodes using elasticsearch output on the metricbeat configuration file).

After setting up this configuration my needs is to keep only these data, logs and metrics only for one week on elasticsearch and on kibana. So i want to keep only the data of the indices for one week and then to be deleted. My indices have the format as filebeat-, winlogbeat- and metricbeat-* and after that they have the version on the beat and the date.

How i can setup a valid lifecycle policy to them and keep only the indices for one week? I tried with the lifecycle policy doc on the elastic site but i couldn't be able to setting up this correctly.

I would be grateful if someone could help me on this.

Best Regards,
Thanos

2

I am also facing this issue. i also tried to setup lifecycle policy but unable to set.

Any input will be helpful.

Thanks.

3

Hi @antonopo and @Tek_Chand
I recommend you to check my solution in this topic about how to configure Index Life cycle Policy:
https://discuss.elastic.co/t/how-to-configure-my-index-life-cycle-policy/193468/3?u=dgonzalezp

This configuration was made to delete indexes older than 10 days, you just have to change it to 7 days and others parameters like index name, index pattern name...
And this configuration will be apllied to the new indexes, the olders ones must be deleted manually.

If you need more help just let me know.

4

Thanks a lot for the reply. After creating the policies i have some errors.
Do you faces something like this?

12
121300×801 79.4 KB

5

What kind of errors show when you click on "Shown errors"

6

13
13.PNG918×699 49.5 KB

7

I am getting the same error to all my indexes that i have added on the policy

8

PUT _ilm/policy/filebeat_policy
{
"policy": {
"phases": {
"delete": {
"min_age": "7d",
"actions": {
"delete": {}
}
}
}
}
}

PUT _template/filebeat_template
{
"index_patterns": ["filebeat-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "filebeat_policy",
"index.lifecycle.rollover_alias": "filebeat"
}
}

9

That was my configuration for filebeat_policy . All my indices filebeat-* were created automatically through logstash.

10

Yes, as I said in my first reply " this configuration will be apllied to the new indexes" the problem is that the old indexes were not created with the template and the old ones dont have alias, for that reason you are getting this error and you have to delete it manually but the new ones will be created with alias and will be deleted automatically when will be older than 7 days.

11

Ok thank you very much. Lets see on the new indices.

12

Well you can try this:

PUT filebeat-*/_settings
{
"settings": {
"index.lifecycle.name": "filebeat_policy",
"index.lifecycle.rollover_alias": "filebeat"
}
}

With that you can update your old indexes and add it the alias of your lifecycle policy, Im not sure if this will work but you can try only with one index.

I just test it and yes your old indexes will be added to your lifecycle policy but I dont know if your index older than 7 days will be deleted automatically or if you have to do it manually but you will not see that errors at least

13

Thanks a lot for your help i think that it worked.

1
11155×285 21.2 KB

14

For Filebeat and Winlogbeat that are sending to logstash was worked perfectly. Every new index has the policy that i have configured.

The problem now is on the new indexes on the metricbeat. Every new index on metricbeat is assigned to a different lifecycle policy of this one that i have created. For example i have created the metricbeat_policy and the new metricbeat indices were assigned to a metricbeat-7.2.0 policy like that. The difference from Metricbeat and Filebeat and Winlogbeat is that has as output the elasticsearch.

That was my settings
PUT _ilm/policy/metricbeat_policy
{
"policy": {
"phases": {
"delete": {
"min_age": "7d",
"actions": {
"delete": {}
}
}
}
}
}
PUT _template/metricbeat_template
{
"index_patterns": ["metricbeat-"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "metricbeat_policy",
"index.lifecycle.rollover_alias": "metricbeat"
}
}
PUT metricbeat-/_settings
{
"settings": {
"index.lifecycle.name": "metricbeat_policy",
"index.lifecycle.rollover_alias": "metricbeat"
}
}

I hade created the same for filebeat and winlogbeat.

This is the

15

You can try deleting the metricbeat-7.2.0 policy

16

I have already deleted the metricbeat-7.2.0 policy.
All the new metricbeat indices that were created today have assigned to this policy

11
11.PNG939×626 40.2 KB
22
22.PNG934×689 41.8 KB
33
33.PNG938×671 41.9 KB
44
44.PNG935×680 41.6 KB

And these are my policies

55
55.PNG997×428 29.8 KB

In order to add the new metricbeat indexes to the metricbeat_polixy i have to tun every day this code:

PUT metricbeat-*/_settings
{
"settings": {
"index.lifecycle.name": "metricbeat_policy",
"index.lifecycle.rollover_alias": "metricbeat"
}
}

And then if you can see they have been added to the metricbeat_policy

66
66.PNG989×439 29.8 KB

The problem is that the new metricbeat indices are not going to metricbeat_policy directly.

17

Have you missed in this template the * symbol in the index_patterns? because that could be the reason why your new indexes are not going to metricbeat_policy

18

These are my settings

PUT _ilm/policy/metricbeat_policy
{
"policy": {
"phases": {
"delete": {
"min_age": "7d",
"actions": {
"delete": {}
}
}
}
}
}
PUT _template/metricbeat_template
{
"index_patterns": ["metricbeat-"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "metricbeat_policy",
"index.lifecycle.rollover_alias": "metricbeat"
}
}
PUT metricbeat-/_settings
{
"settings": {
"index.lifecycle.name": "metricbeat_policy",
"index.lifecycle.rollover_alias": "metricbeat"
}
}

19

77
77.PNG673×552 10.2 KB

When i copy paste it on the reply window the * was deleted but as you see my screenshot i already mave it as metricbeat-*

20

Hello,

This link (use google to translate) has helped me a lot to use ILM policy