See you are deploying on a Kubernetes cluster You could try to add an index filed to the output like this index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}", see the documentation here: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html. Depending on your use case, you should also consider if you want a new index created every day. The recommended index size is around 50GB
*ignore the face that it has 28 and 29 July, its created by ILM
you mentioning about 50GB index size, is ILM what you means? if thats what you mean, i did try that, but since my cluster only live for office hours, the lifetime of an index is not 24 hours, so when i try ILM with 1 day Max age, it wont generate new index in the next day (since the cluster age is not 24 hours yet).
so, is filebeat creating new index based on ILM only, or its creating new index by output configuration? since if its ILM, then i dont have any way to create it every dat then
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.