I want to replicate with filebeat does, create a new index daily. I just want to rollover on the event of unusual data volume. Say I normally get 20G a day, so I set 1 shard. I want to rollover if I get more than 50G any one day.
It seems to work with for the first day, but it didn't roll over, it showed an error in Kibana index management, something like "index.lifecycle.rollover_alias" not defined. That IS defined in the logstash output, but it wasn't in the my-index-dd.yy.mm-000001 settings. I added it, then rollover happened. (So I can't get the exact error message back).
Do I need to add this field to the template for my-index? The rollover_alias isn't in the -00002 index yet either, so I anticipate the same error at the next rollover.
Do I need a different alias? I just set it to "my-index" to try to keep it simple.
Do I need to add this field to the template for my-index?
Yes, your index template for the my-index* indices should have the setting index.lifecycle.rollover_alias set, similar to the example in this documentation.
I found it a bit odd that you would need to do this if you have the rollover alias set in Logstash, but there is a note in the Elasticsearch output docs that looks like it might be saying you need to do this:
If the rollover alias or pattern is modified, the index template will need to be overwritten as the settings index.lifecycle.name and index.lifecycle.rollover_alias are automatically written to the template
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.