.ds-.... creation is driven by the ILM policy and the backing indices are meant to be opaque... if ILM is not set daily then new .ds-... will not be created daily
Since your config is creating a new datastream every day (which you do not want to do) then you get a backing index every day
But filebeat do this by default. It creates one datastream per day.
When I say "ds", I mean "datastream". Is there anyway to create only one datastream instead of creating one each day? WIth my current config, a new datastream is created everyday, as well as a index.
I want to create only one datastream and then all the indicies create by filebeat to be a part of that datastream. Is this possible to achieve?
No... it is because you set the index name like this index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd} ..........................................^^^^^^^^^^^^^^^ <!- THIS is not correct
That says create a new datastream every day....
Set the index as I suggested and it will not
If you do not set the index name at all it will create a datastream
filebeat-8.7.0 no daily date
Filebeat does not create a data stream per day by default, its is doing it because you configured it to it.
Try what I suggested first... your understanding is not correct
index: voucher-app-logs-%{[agent.version]}
THEN you can control how often the underlying .ds-.... is created with the ILM policy
Wow, that's nice. Then the datastream created by filebeat follows the pattern described in the index parameter?
I haven't found that in the docs, perhaps I missed something, will check that later. I thought that the index parameter was intended to configure the index name created, even when ilm was enabled.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.