Custom filebeat index - same index name but diferent template

[bueka.torao (Bueká Torao)]

Hi guys,

I am setting my filebeat.yml in order to send logs to Elasticsearch. I created a template in Elasticsearch, I named it: filebeat . See below the configuration.

I set the filebeat.yml as you guys can see below.

The issue is that filebeat keeps creating index like this:
filebeat-7.1.1-2020.02.21-000001

Why is adding ".21-000001" at then end of the index name?

Thanks in advance.

Filebeat.yml
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:

Array of hosts to connect to.

hosts: ["10.206.13.168:9200","10.206.13.169:9200"]
index: "filebeat-%{[beat.version]}-%{+yyyy.MM}"
setup.template:
name: "filebeat"
pattern: "filebeat-*"
enable: false

Template:
{
"filebeat" : {
"order" : 1,
"index_patterns" : [
"filebeat-*"
],
"settings" : {
"index" : {
"routing" : {
"allocation" : {
"total_shards_per_node" : "5"
}
},
"mapping" : {
"total_fields" : {
"limit" : "3000"
}
},
"refresh_interval" : "20s",
"number_of_shards" : "2",
"auto_expand_replicas" : "false",
"requests" : {
"cache" : {
"enable" : "true"
}
},
"number_of_replicas" : "1"
}
},
"mappings" : {
"dynamic_templates" : [
{
"integers" : {
"mapping" : {
"type" : "integer"
},
"match_mapping_type" : "long"
}
},
{
"strings" : {
"mapping" : {
"type" : "keyword"
},
"match_mapping_type" : "string"
}
}
],
"properties" : {
"geo.location" : {
"type" : "geo_point"
},
"tw_seguidores" : {
"type" : "long"
}
}
},
"aliases" : { }
}
}

[jsoriano (Jaime Soriano)]

Hey @bueka.torao,

Why are you creating your own index template for Filebeat? It shouldn't be needed for most of the cases.

In any case, if you need to do it, take into account that since Beats 7.0 indexes are managed by ILM. If you want to manage indexes and templates on your own you will need to disable ILM. You can read more about ILM configuration here: https://www.elastic.co/guide/en/beats/filebeat/7.1/ilm.html

[bueka.torao (Bueká Torao)]

Hi @jsoriano, what I really want to do is avoiding Filebeat to create one index for each day. So, I want to have one filebeat index per month. Something like filebeat-7.1.1-2020-02 instead of filebeat-7.1.1-2020-02-01-00001

I am going to check the ILM configuration that you just shared with me in order to understand it better. Thanks!

[bueka.torao (Bueká Torao)]

I solved it using this configuration, I don´t know if it is the best configuration.

#==================== Elasticsearch template setting ==========================

setup.template.settings:
name: "filebeat"
pattern: "filebeat-*"
enable: false

setup.ilm.enabled: true
setup.ilm.pattern: "{now/M-1M{yyyy.MM}}"

[jsoriano (Jaime Soriano)]

ILM configuration doesn't create an index per day by default in Filebeat, it creates one new index when one of this happens:

• Current index is more than 30 days old
• Current index takes more than 50GB

This uses to be a sane default for most of the cases. It is not recommended to have very big indexes, this is why the size limit is added. And having more than a month of logs in an index can complicate the deletion of old data.

With default configuration, if more than 50GB are logged per day, then there can be multiple indexes with the same day (e.g. filebeat-7.1.1-2020-02-01-00001, filebeat-7.1.1-2020-02-01-00002 and so on).

With your configuration, if more than 50GB are logged per month, there will be also multiple indexes for the same month, but with an incremented number in the postfix. Take into account that this pattern is only used in the moment where the index is created, so the timestamp you see there is only the day when the index was created, but it can contain data of the following days.

If you want to modify the default policies, you can do it from Kibana, or using the API.

I think that for your case the default management policies will work for you, because they will create an index every 30 days, unless you have more than 50GB per month, but in this case it would be also recommended to use more than one index.

[bueka.torao (Bueká Torao)]

Hi @jsoriano, ok. I am getting it now. You are right, the defaults configuration will work for me then as you mentioned.
My misunderstanding was the name of the index, however, since the timestamp is used only the day when the index was created, and it contains data of the following days, my issue is solved.

Thanks!!!