# ILM failing for rollover for Wazuh module / data
**URL:** https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781
**Category:** Elasticsearch
**Tags:** ilm-index-lifecycle-management
**Created:** [December 14, 2021, 11:10am UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781 "2021-12-14T11:10:19Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)
#### Post date: [December 14, 2021, 11:10am UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/1 "2021-12-14T11:10:19Z")
I have setup a rollover policy for wazuh index with the below policy defination
```auto
{
"policy": "Wazuh",
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "20gb",
"max_age": "1d"
}
}
},
"version": 4,
"modified_date_in_millis": 1639189543104
}
```
Per my understanding there are 3 basic requirements as mentioned in below link
> **[Tutorial: Automate rollover with ILM | Elasticsearch Guide \[7.10\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/getting-started-index-lifecycle-management.html)**
I have create a IML policy as mentioned below
```auto
PUT _ilm/policy/Wazuh
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "20gb",
"max_age": "1d"
}
}
},
"warm": {
"actions": {}
},
"delete": {
"min_age": "365d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}
```
Created a template to associate with index pattern
```auto
{
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "Wazuh",
"rollover_alias": "wazuh-alerts"
},
"number_of_shards": "3",
"number_of_replicas": "1"
}
},
"aliases": {},
"mappings": {}
}
}
```
Created a inital write index with name " wazuh-alerts-000001"
I am seeing index getting as large as 90G for single day and the below error in rollover
```auto
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [wazuh-alerts] does not point to index [wazuh-alerts-4.x-2021.12.13]
at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:114)
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:174)
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:327)
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:265)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:216)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
```
What am i missing ?
---
### Author:  [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)
#### Post date: [December 15, 2021, 2:28am UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/2 "2021-12-15T02:28:04Z")
Checking if anyone has any ideas?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [December 15, 2021, 3:44am UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/3 "2021-12-15T03:44:20Z")
Perhaps You forgot to create the initial managed index correctly with the write alias
> **[Configure a lifecycle policy | Elasticsearch Guide \[7.16\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html#create-initial-index)**
You should have done this...
```auto
PUT wazuh-alerts-4.x-2021.12.13
{
"aliases": {
"wazuh-alerts":{
"is_write_index": true
}
}
}
```
Then what you are using to index should point to the write alias `wazuh-alerts` instead looks like you are writing directly to the concrete index `wazuh-alerts-4.x-2021.12.13`
---
### Author:  [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)
#### Post date: [December 15, 2021, 7:46am UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/4 "2021-12-15T07:46:12Z")
Thanks @stephenb for the response. I tried the suggested approach however didnt had much luck.
Here are some other changes i tried
Filebeat.yml
```auto
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
#setup.ilm.overwrite: true
setup.ilm.rollover_alias: "wazuh-alerts-4.x"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.enabled: true
setup.ilm.policy_name: 'Wazuh'
setup.ilm.policy_file: '/etc/filebeat/policy.json'
```
I see a write index was created when i restarted filebeat on wazuh manager
I see an alias on the write index however none on the other index created for the day
Do we need the index to be in format wazuh-alerts-4.x-2021.12.15-000001 where 000001 will rollover to 000002 and so on for today date ?
If so, how can i force creation of the index with such naming convention ?
---
### Author:  [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)
#### Post date: [December 15, 2021, 1:45pm UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/5 "2021-12-15T13:45:48Z")
I have tried creating a write index again with name "wazuh-alerts-4.x-2021.12.13-000001" and on rollover i see its trying to rollover the write index itself rather than my main index in question wazuh-alerts-4.x-2021.12.15.
Seems like some misconfiguration to me.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [December 15, 2021, 2:41pm UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/6 "2021-12-15T14:41:05Z")
Can you show the output section in your filebeat.yml.
The index you're writing to should be `wazuh-alerts-4.x` the alias name.
Also that should be the alias in your first managed index at all has to line up.
```auto
PUT wazuh-alerts-4.x-2021.12.13-000001
{
"aliases": {
"wazuh-alerts-4.x":{
"is_write_index": true
}
}
}
```
It all has to line up.
Also I just noticed this.
Get rid of all this.. You're putting the template in by hand in the dev tools I don't know what's in all these but you're probably overriding and that's causing those weird templates.
---
### Author:  [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)
#### Post date: [December 15, 2021, 3:38pm UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/7 "2021-12-15T15:38:06Z")
@stephenb Thank you for the update, i tried to update the alias and recreated the index to match todays date.
I am getting a different error this time "` "reason" : "index name [wazuh-alerts-4.x-2021.12.15] does not match pattern '^.*-\\d+$'"`
I see similar issue raised in the below post.
> [@Using rollover got error](https://discuss.elastic.co/t/using-rollover-got-error/84869):
>
> Hi, I just created an alias which name is "squid-log-alias" for my index "squid-log", and when I setup rollover API and I got the error: POST /squid-log-alias/\_rollover { "conditions":{ "max\_age": "7d", "max\_docs": 1000 } } { "error": { "root\_cause": [{ "type": "illegal\_argument\_exception", "reason": "index name [squid-log] does not match pattern '^.-\d+$'" } ], "type": "illegal\_argument\_exception", "reason": "index name [squid-log] does not match pattern '^.-\d+$'" }, "staβ¦
Do we need the increment number after the index to rollover ? I am fine to clear out the settings and start fresh if that helps.
---
### Author:  [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)
#### Post date: [December 15, 2021, 3:41pm UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/8 "2021-12-15T15:41:29Z")
My current filebeat.yml is
```auto
output.elasticsearch:
hosts: [00000]
protocol: http
username: "00000"
password: "00000"
ssl.certificate_authorities:
- /etc/filebeat/certs/root-ca.pem
ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
ssl.key: "/etc/filebeat/certs/filebeat-key.pem"
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.ilm.overwrite: true
setup.ilm.enabled: false
setup.ilm.rollover_alias: "wazuh-alerts-4.x"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.enabled: true
setup.ilm.policy_name: 'Wazuh'
setup.ilm.policy_file: '/etc/filebeat/policy.json'
filebeat.modules:
- module: wazuh
alerts:
enabled: true
archives:
enabled: false
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [December 15, 2021, 3:55pm UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/9 "2021-12-15T15:55:47Z")
I fixed above... for initial managed
```auto
PUT wazuh-alerts-4.x-2021.12.13-000001
{
"aliases": {
"wazuh-alerts-4.x":{
"is_write_index": true
}
}
}
```
Think you need...
```auto
output.elasticsearch:
hosts: [00000]
index: wazuh-alerts-4.x
```
> [@Atul\_Chadha](#):
>
> Do we need the increment number after the index to rollover ? I am fine to clear out the settings and start fresh if that helps.
Not sure I quite understand ... No you should not increment that should happen automatically.
If you are saying the you need to go to ....000002 then you need to set the old index to `"is_write_index": false` and create a new managed index.
Not sure why you are doing all this...
```auto
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.ilm.policy_name: 'Wazuh'
setup.ilm.policy_file: '/etc/filebeat/policy.json'
```
Once you set it via Dev Tools or UI.. you don't need this.
I would set.
```auto
setup.ilm.overwrite: false
```
Also ... I would think it would be
```auto
setup.ilm.enabled: false
```
If the policy is set in the template you don't need to set it here.
Seems like you are setting all sorts of settings.....
BTW Also this is why we really have people just **start with the defaults** (filebeat index) get used to it all .. get it working with the defaults and **THEN** change to custom index names etc.. etc..
If you are following some other tutorial...perhaps take another read...
---
### Author:  [@Atul\_Chadha](https://discuss.elastic.co/u/Atul_Chadha)
#### Post date: [December 16, 2021, 4:27pm UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/10 "2021-12-16T16:27:48Z")
I think there may be a bug in what we are trying to achieve ([[Docs] Note that output.elasticsearch.index setting is ignored with ILM enabled Β· Issue #11866 Β· elastic/beats Β· GitHub](https://github.com/elastic/beats/issues/11866))
@stephenb Appreciate your efforts and expertise. Will try to see what other option i can try.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [December 16, 2021, 4:34pm UTC](https://discuss.elastic.co/t/ilm-failing-for-rollover-for-wazuh-module-data/291781/11 "2021-12-16T16:34:40Z")
@Atul_Chadha
Perhaps but I don't think so ... that is against and old version.
I configure custom indexes with ILM with filebeat every day...
The Problem is ... It is hard from me to follow everything you are doing (or not doing)... we just set up 8 custom indices with ILM yesterday for a customer using filebeat it is running today and rolling over just fine.
The basic Steps:
Create and ILM Policy
Create a Template with the index patterns, the ILM policy and the rollover alias.
Create the initial managed index
Put the minimal required config in filebeat (This is where people struggle)
Run Filebeat.
AND you can always just use the defaults and get it running π
Example here is a complete filebeat where we setup the ILM manually notice how little is there
```auto
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
β
filebeat.inputs:
- type: aws-s3
enabled: true
queue_url:
tags: [forwarded, custom-logs]
index: "custom-logs-writer-alias"