As you can see in the image the part in red box what i want to know is where is that file for template resides i want to see that configuration as i want to made the custom template for that sake need a help for configuration from there.
Hello!
You can find the procedure to export the template in our doc in this section
thanks @Luca_Belluccini ......
if i am not wrong this is the one which will be used by the filebeat
but it doesn't contain any alias even though i read that ILM policy will be used by defualt in elasitc 7.x ........ and in order to roll over you need to have the alias .... can you tell me from where and how does it will be able to get that alias for the template
The alias is created at Filebeat startup automatically.
If I correctly understand, your only objective is to make Filebeat work as usual, but customize the destination index template.
The exported index template should contain the following settings:
"settings": {
"index": {
"lifecycle": {
"name": "filebeat",
"rollover_alias": "filebeat-7.6.2"
},
Which means it will make use of the filebeat ILM policy and it will require filebeat-7.6.2 alias.
The rollover alias gets created when Filebeat starts for the first time.
It should not require any change on the output.elasticsearch section (except the connection parameters).
My suggestion is:
- Export the Index Template
- Customize it
- Install it to the Elasticsearch cluster
- On
filebeat.yml, set the following (so if you will upgrade Filebeat, the customized Index Pattern will not be overwritten)setup.template.enabled: falsesetup.template.overwrite: false
- Start Filebeat. It should create an alias
filebeat-7.6.2and write to it
Please note on every Filebeat update, we might introduce changes to the Index Template.
So please repeat this procedure each time you upgrade.
thanks @Luca_Belluccini i followed your advice that is really valuable but now i'm facing some other issue. I am testing open source wazuh with elasticsearch which create daily time based indices what i'm trying to do is implement the index life cycle management on the index template for filebeat which comes with wazuh. But the problem i'm facing is this
as we know that the index behind the alias which is used for rollover should end with the incremental value such as 000001. but the index created by the index template ending is based on this attribute which is date not an incremental value so i'm getting this error that index name should end with the incremental value
"index_name_format": "yyyy.MM.dd","
but i tired adding 00001 in the end in such way
"index_name_format": "yyyy.MM.dd-000001",
ILM create new index on rollover of with following format which is fine
"yyyy.MM.dd-000002"
but my filebeat is still looking for yyyy.MM.dd-000001 even though it should right to the 000002 can you help me with this... how should i modify the
"index_name_format": "yyyy.MM.dd","
so filebeat start righting to the next generated index on rollover instead of waiting for the previous one....
Sorry, I do not know about Wazuh, but I can try to help.
I do not know where the setting index_name_format is set as this setting doesn't exist in Filebeat.
Where does this variable ends up?
I searched here but I could not find any match.
The concept is Filebeat, when ILM is enabled, has to write to <rollover alias>.
The <rollover alias> is bootstrapped by Filebeat if ILM is enabled (also consider Elasticsearch must have at least a basic license) creating an index with a alias (with is_write_index: true), automatically. E.g. <rollover alias> points to <index prefix>-yyyy.MM.dd-000001
that line is in ingest pipeline in their Filebeat module whose path is this /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json in that file you can see this processor which by default creates indices on a daily basis:
{
"date_index_name": {
"field": "timestamp",
"date_rounding": "d",
"index_name_prefix": "{{fields.index_prefix}}",
"index_name_format": "yyyy.MM.dd",
"ignore_failure": false
}
},
if you want to change the name or the duration after which indices should create, you can simply edit this param
"index_name_format": "yyyy.MM.dd",
As far as license is concerned i do have the basic licenese as you can see
Unfortunately there are few things to address.
The module written by wazuh seems not compatible with how ILM & rollover works.
There are 2 main reasons:
- It creates multiple destination indices as
fields.index_prefixmight change depending on the incoming event - the
index_name_formatseems to require a date rollover
I didn't test it but I think the only way to make this wazuh module compatible with ILM would be to:
- bootstrap all the destination indices manually
- set
index_name_formatto""(if possible)
Do you know all the possible values of index_prefix?
yes fields.index_prefix it doesn't change it's remain same/static always that is wazuh-alerts-3.x-
{
"indices" : {
"wazuh-alerts-3.x-2020.04.26.00" : {
"index" : "wazuh-alerts-3.x-2020.04.26.00",
"managed" : true,
"policy" : "wazuh_policy",
"lifecycle_date_millis" : 1587907263243,
"age" : "6.72m",
"phase" : "hot",
"phase_time_millis" : 1587907648619,
"action" : "rollover",
"action_time_millis" : 1587907272222,
"step" : "ERROR",
"step_time_millis" : 1587907658572,
"failed_step" : "check-rollover-ready",
"is_auto_retryable_error" : true,
"failed_step_retry_count" : 19,
"step_info" : {
"type" : "illegal_argument_exception",
"reason" : """index name [<wazuh-alerts-3.x-{2020.04.26.20||/s{yyyy.MM.dd.mm|UTC}}>] does not match pattern '^.*-\d+$'""",
"stack_trace" : """java.lang.IllegalArgumentException: index name [<wazuh-alerts-3.x-{2020.04.26.20||/s{yyyy.MM.dd.mm|UTC}}>] does not match pattern '^.*-\d+$'
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.generateRolloverIndexName(TransportRolloverAction.java:236)
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:131)
this is the exception i'm getting
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.