ILM policy being over written

Elastic Stack Elasticsearch
1

Hi

This is a basic question. But can you please clarify.

I have a setting in my file beat configuration.

setup.ilm.enabled: auto

# Configure the ILM write alias name.
setup.ilm.rollover_alias: "metricbeat"

# Configure rollover index pattern.
setup.ilm.pattern: "{now/d}-000001"

I am installing beats on multiple server. I see with every installation the version of the ILM policy is changing. So I assume its being over written.

image
image.png1198×176 11.8 KB

So I tried to change the setting of this from setup.ilm.enabled: auto to setup.ilm.enabled: false. But the installation is failing if I do this.

I also manually changed the configuration of ILM to delete the index after 1 year but its changing to 30 days.

Can you please tell me how I can just set this once and make sure that is not over written with every installation of beats agent.

2

I am just sharing what i have done.
I overwrite it with the same config i set each time i run filebeat setup -e

You can read up on setup.ilm.check_exists: & setup.ilm.overwrite: settings in the docs.

setup.ilm.enabled: true

setup.ilm.policy_name: "filebeat-7.0.1"

setup.ilm.policy_file: "/etc/filebeat/filebeat-7.0.1.json"

setup.ilm.check_exists: false

setup.ilm.overwrite: true

setup.ilm.rollover_alias: "filebeat-linuxclient"

setup.ilm.pattern: "{now/d}-000001"

My filebeat-7.0.1.json file:

{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "rollover": {
                        "max_age": "1d",
                        "max_size": "15gb"
                    }
                }
            },
            "warm": {
                "min_age": "30d",
                "actions": {}
            },
            "cold": {
                "min_age": "60d",
                "actions": {}
            },
            "delete": {
                "min_age": "90d",
                "actions": {
                    "delete": {}
                }
            }
        }
    }
}

With this, even if it overwrites, it will overwrite with the policy you have set. However, you need to rollover the indices manually for them to be in sync to the same version.

1 Like
3

@TsuWeiQuan Thanks for your reply. Though your settings work. I believe we must not overwrite the settings which is set once.

It is better if we manage this setting from one place rather than overwriting every time when we install beats agent.

I can set this from the admin settings page. But I want to remove this from beats configuration.

4

Yes, you are right. You can set
setup.ilm.check_exists: true
setup.ilm.overwrite: false
to disable overwrite too. I recalled i had some issue with it hence i did it the overwrite method for the time being

2 Likes
5

Thank you. Let me try this setting.

6 
setup.ilm.check_exists: true
setup.ilm.overwrite: false

I presume that there is a bug in these settings.
Even tho ilm overwrite was set to false. Running filebeat setup -e would still overwrite the policy.
Similar scenario appears in metricbeat too.

2019-07-04T11:15:00.216+0800    INFO    [index-management]      idxmgmt/std.go:400      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-cei2sitsvr {now/d}-000001} as ILM is enabled.
2019-07-04T11:15:00.216+0800    INFO    [index-management]      idxmgmt/std.go:404      Set settings.index.lifecycle.name in template to {metricbeat-7.0.1 map[policy:map[phases:map[cold:map[min_age:3d actions:map[]] delete:map[min_age:5d actions:map[delete:map[]]] hot:map[min_age:0ms actions:map[rollover:map[max_age:1dmax_size:15gb]]] warm:map[min_age:3d actions:map[]]]]]} as ILM is enabled.
7

@Kaiyan_Sheng Can you please help with this issue or redirect us to the right person. Thank you.

8

@TsuWeiQuan I tried the settings and Yes it does not work and overwrite the ILM policy settings.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.