Lots of great questions. Let me try to give you some pointers.
ILM is typically used in combination with rollovers. The rollover API causes new indexes to be created continuously. Initially, all of your data goes into an index called something like logs-000001
. After that index reaches a certain age or a certain size, it "rolls over" to a new index logs-000002
and all new data will go to that new index. After a while, that index will rollover to a new index logs-000003
etc etc.
Now, your applications (Beats) do not need to know what the name if the current active index is. That's because all these indexes will sit behind an alias (logs-alias
in the diagram below). When querying the data, all you need to query is that alias, and the query will automatically hit the underlying indexes. When you write data, you write the data to that alias, and the data will end up in the active index. This works because the alias will have one "write index", which is always pointing to the current active index. When an index rolls over from my_index-000002
to my_index-000003
, the write index is automatically adjusted.
The error you were seeing has to do with the alias. It was not properly set up. No write index had been configured. This would cause problems indexing the data, because Elasticsearch would not know what index to write the documents to. It also caused problems with the rollover API. An index can only rollover if it is the current write index for an alias that's pointing to multiple indexes.
To prevent those problems, don't configure aliases in the index template. It's fine to use index templates for settings and mappings, but not for the alias. You configure the alias when you create the first index. After that, ILM will take care of adding new indexes to the alias and flipping the write index for you.