Created a inital write index with name " wazuh-alerts-000001"
I am seeing index getting as large as 90G for single day and the below error in rollover
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [wazuh-alerts] does not point to index [wazuh-alerts-4.x-2021.12.13]
I have tried creating a write index again with name "wazuh-alerts-4.x-2021.12.13-000001" and on rollover i see its trying to rollover the write index itself rather than my main index in question wazuh-alerts-4.x-2021.12.15.
Also I just noticed this.
Get rid of all this.. You're putting the template in by hand in the dev tools I don't know what's in all these but you're probably overriding and that's causing those weird templates.
Not sure I quite understand ... No you should not increment that should happen automatically.
If you are saying the you need to go to ....000002 then you need to set the old index to "is_write_index": false and create a new managed index.
Perhaps but I don't think so ... that is against and old version.
I configure custom indexes with ILM with filebeat every day...
The Problem is ... It is hard from me to follow everything you are doing (or not doing)... we just set up 8 custom indices with ILM yesterday for a customer using filebeat it is running today and rolling over just fine.
The basic Steps:
Create and ILM Policy
Create a Template with the index patterns, the ILM policy and the rollover alias.
Create the initial managed index
Put the minimal required config in filebeat (This is where people struggle)
AND you can always just use the defaults and get it running
Example here is a complete filebeat where we setup the ILM manually notice how little is there
- type: aws-s3
tags: [forwarded, custom-logs]
index: "custom-logs-writer-alias" <!---- KEY as we set it up all manually with the template, ILM etc
setup.ilm.enabled: false <!---- KEY as we set it up all manually with the template, ILM etc
The above I just posted used a custom template... you are close... don't give up... you are over configuring filebeat and it is conflicting with what you setup with the Template, ILM, and Initial Managed Index with the Writer Alias
Create and ILM policy
Just load the Wazuh Template... and add that policy to it.
Create the initial managed index
Then use minimal filebeat like above.
the setup.ilm.enabled: false
Seems counter intuitive.. what this says is filebeat don't manage the ILM I have set it up manually
2021-12-16T12:59:56.801-0500 INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-12-16T12:59:56.802-0500 DEBUG [beat] instance/beat.go:697 Beat metadata path: /var/lib/filebeat/meta.json
2021-12-16T12:59:56.802-0500 INFO instance/beat.go:653 Beat ID: 9e4203d0-9719-48b9-8970-d4f876665e6f
2021-12-16T12:59:56.802-0500 INFO instance/beat.go:373 filebeat stopped.
2021-12-16T12:59:56.806-0500 ERROR instance/beat.go:956 Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified
Yes tiny policies are not going to work the way you think...
1m / 20 mb is not really valid it will roll over but tiny fast / will not be exact.. it will rollover when it gets to it... background task. That will work right at scale of GBs / Day
BUT it looks like you did not create the initial managed index correct it should be pattern
Delete all the indices
Go to Kibana Dev Tools and do the following