I am setting my filebeat.yml in order to send logs to Elasticsearch. I created a template in Elasticsearch, I named it: filebeat . See below the configuration.
I set the filebeat.yml as you guys can see below.
The issue is that filebeat keeps creating index like this:
filebeat-7.1.1-2020.02.21-000001
Why is adding ".21-000001" at then end of the index name?
Why are you creating your own index template for Filebeat? It shouldn't be needed for most of the cases.
In any case, if you need to do it, take into account that since Beats 7.0 indexes are managed by ILM. If you want to manage indexes and templates on your own you will need to disable ILM. You can read more about ILM configuration here: https://www.elastic.co/guide/en/beats/filebeat/7.1/ilm.html
Hi @jsoriano, what I really want to do is avoiding Filebeat to create one index for each day. So, I want to have one filebeat index per month. Something like filebeat-7.1.1-2020-02 instead of filebeat-7.1.1-2020-02-01-00001
I am going to check the ILM configuration that you just shared with me in order to understand it better. Thanks!
ILM configuration doesn't create an index per day by default in Filebeat, it creates one new index when one of this happens:
Current index is more than 30 days old
Current index takes more than 50GB
This uses to be a sane default for most of the cases. It is not recommended to have very big indexes, this is why the size limit is added. And having more than a month of logs in an index can complicate the deletion of old data.
With default configuration, if more than 50GB are logged per day, then there can be multiple indexes with the same day (e.g. filebeat-7.1.1-2020-02-01-00001, filebeat-7.1.1-2020-02-01-00002 and so on).
With your configuration, if more than 50GB are logged per month, there will be also multiple indexes for the same month, but with an incremented number in the postfix. Take into account that this pattern is only used in the moment where the index is created, so the timestamp you see there is only the day when the index was created, but it can contain data of the following days.
If you want to modify the default policies, you can do it from Kibana, or using the API.
I think that for your case the default management policies will work for you, because they will create an index every 30 days, unless you have more than 50GB per month, but in this case it would be also recommended to use more than one index.
Hi @jsoriano, ok. I am getting it now. You are right, the defaults configuration will work for me then as you mentioned.
My misunderstanding was the name of the index, however, since the timestamp is used only the day when the index was created, and it contains data of the following days, my issue is solved.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.