ELK rollover renamed indexes

Elastic Stack Elasticsearch
1

Hi, I'm using ELK stack version 7.10.1 ( elasticsearch, kibana, filebeat )
My goal is to apply lifecycle policy to my indexes with ILM. I remaned indexes, it's something like this:
filebeat-7.10.1-2021.04.06-000001 ( generated by default with ILM)
apache-2021.04.06-000001
audit-2021.04.06-000001
system-2021.04.06-000001
Fistly I'm trying to configure rollover for apache logs (apache module is enabled).

I created Policy, template and first index with is_write_index option:
#create ilm policy
PUT /_ilm/policy/apache_policy_test
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "2mB",
"max_docs": 100,
"max_age": "1d"
}
}
},
"delete": {
"min_age": "3m",
"actions": {
"delete": {}
}
}
}
}
}

create template

PUT _template/apache_template_test
{
"index_patterns": ["apache-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "apache_policy_test",
"index.lifecycle.rollover_alias": "apache_rollover_alias_test"
}
}

################# filebeat config
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat.log
keepfiles: 7
permissions: 0644

filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • /var/log/*.log
  • type: filestream
    enabled: false
    paths:
    • /var/log/*.log

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

setup.template.settings:
index.number_of_shards: 1

setup.kibana:
host: "192.168.1.156:5601"

output.elasticsearch:
hosts: ["192.168.1.156:9200"]
indices:
- index: "apache-%{+yyyy.MM.dd}-000001"
when.equals:
event.module: "apache"

processors:

  • add_host_metadata:
    when.not.contains.tags: forwarded
  • add_cloud_metadata: ~
  • add_docker_metadata: ~
  • add_kubernetes_metadata: ~
    ################# END

Apache index has ' Aliases none ', so no rollover
error message:
"illegal_argument_exception: index.lifecycle.rollover_alias [apache_rollover_alias_test] does not point to index [apache-2021.04.06-000001] "
I retried differently by creating the first index. The rollover works but the next index apache-2021.04.06-000002 doesn't updated (doc count is always 0)

create first index with aliases

PUT /apache-2021.04.06-000001
{
"aliases": {
"apache_rollover_alias_test": {
"is_write_index": true
}
}
}

get index informations

GET apache-2021.04.06-000001/_ilm/explain

retry the policy

POST apache-2021.04.06-000001/_ilm/retry

Finally nothing works.
what is wrong? have I forgotten something?
Thanks you !

2

Welcome to our community! :smiley:

If you created the apache-2021.04.06-000001 index with the apache_rollover_alias_test alias, what was the output from GET apache-2021.04.06-000001/_ilm/explain?
You shouldn't need to run a retry on the policy though, if you just created it then it's unlikely to fill your rollover criteria.

3

Hi @warkolm ,
I used GET just to check the created index (it's not really useful)
Why retry policy? Maybe it's useless or that's not the way to do it.
I also tried whitout retry policy.
The next index ( number 2) is not updated (but has a rollover alias) and the first index has no more aliase (Alias None), so I got the error ( rollover_alias does not point to index )
Thanks

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.