Elasticsearch ILM pattern suffix

Hi,

I'd like to find a way to "reset" the ILM pattern suffix number. My indexes are written as follows:

filebeat-2019.09.16-000034
filebeat-2019.09.16-000035
filebeat-2019.09.16-000036
filebeat-2019.09.16-000037
filebeat-2019.09.16-000038
filebeat-2019.09.17-000039
filebeat-2019.09.17-000040
filebeat-2019.09.17-000041
filebeat-2019.09.17-000042
filebeat-2019.09.17-000043

I'd like the suffix to reset back to 000001 when the new day is written. Is this possible? I'm using Filebeat and Logstash with the following configurations:

Logstash Output
==============
output {
    elasticsearch {
        hosts => my_elasticsearch_hosts
        ilm_enabled => true
        ilm_rollover_alias => "%{[@metadata][beat]}"
        ilm_pattern => "{now/d}-000001"
        ilm_policy => "filebeat_ilm"
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  	}
}

&

Filebeat.yml
#============================== Template =====================================
    setup.template.enabled: true
    setup.template.fields: ${path.config}/fields.yml
    setup.template.name: filebeat
    setup.template.overwrite: true
    setup.template.pattern: filebeat-*
    setup.template.settings:
        index:
            codec: best_compression
            number_of_replicas: 2
            number_of_shards: 3
            routing.allocation.require.data: hot
            shard.check_on_startup: checksum

#============================== Setup ILM =====================================
    setup.ilm.check_exists: false
    setup.ilm.enabled: auto
    setup.ilm.overwrite: true
    setup.ilm.pattern: '{now/d}-000001'
    setup.ilm.policy_file: /etc/filebeat/ilm_policy.json
    setup.ilm.policy_name: filebeat_ilm
    setup.ilm.rollover_alias: filebeat

&

ilm_policy.json
    {"policy": {
            "phases": {
                "hot": {
                    "min_age": "0ms",
                    "actions": {
                        "rollover": {
                            "max_age": "30d",
                            "max_size": "100mb"
                        }
                    }
                },
                "warm": {
                    "min_age": "90d",
                    "actions": {
                        "allocate": {
                            "include": {},
                            "exclude": {},
                            "require": {
                                "data": "warm"
                            }
                        }
                    }
                }
            }
        }
    }

The ILM policy is only set at 100mb for testing, this will be set to ~30GB when live.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.