Hi,
I'd like to find a way to "reset" the ILM pattern suffix number. My indexes are written as follows:
filebeat-2019.09.16-000034
filebeat-2019.09.16-000035
filebeat-2019.09.16-000036
filebeat-2019.09.16-000037
filebeat-2019.09.16-000038
filebeat-2019.09.17-000039
filebeat-2019.09.17-000040
filebeat-2019.09.17-000041
filebeat-2019.09.17-000042
filebeat-2019.09.17-000043
I'd like the suffix to reset back to 000001 when the new day is written. Is this possible? I'm using Filebeat and Logstash with the following configurations:
Logstash Output
==============
output {
elasticsearch {
hosts => my_elasticsearch_hosts
ilm_enabled => true
ilm_rollover_alias => "%{[@metadata][beat]}"
ilm_pattern => "{now/d}-000001"
ilm_policy => "filebeat_ilm"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
}
}
&
Filebeat.yml
#============================== Template =====================================
setup.template.enabled: true
setup.template.fields: ${path.config}/fields.yml
setup.template.name: filebeat
setup.template.overwrite: true
setup.template.pattern: filebeat-*
setup.template.settings:
index:
codec: best_compression
number_of_replicas: 2
number_of_shards: 3
routing.allocation.require.data: hot
shard.check_on_startup: checksum
#============================== Setup ILM =====================================
setup.ilm.check_exists: false
setup.ilm.enabled: auto
setup.ilm.overwrite: true
setup.ilm.pattern: '{now/d}-000001'
setup.ilm.policy_file: /etc/filebeat/ilm_policy.json
setup.ilm.policy_name: filebeat_ilm
setup.ilm.rollover_alias: filebeat
&
ilm_policy.json
{"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "30d",
"max_size": "100mb"
}
}
},
"warm": {
"min_age": "90d",
"actions": {
"allocate": {
"include": {},
"exclude": {},
"require": {
"data": "warm"
}
}
}
}
}
}
}
The ILM policy is only set at 100mb for testing, this will be set to ~30GB when live.