I'm about to transition our log collection fleet to using ILM. Looks promising, but I have a question.
Currently we use time (by day) based indices. Filebeat sends to localhost logstash, and logstash bulk indexes to Elasticsearch. With this setup is relatively easy to know which index a log will appear in - it's based on "@timestamp".
However with ILM the new index is rolled after a given size or time period. It also looks like older indices can be marked as read only, shrunk, and merged. This is a great feature.
However, what happens when a specific node hypothetically loses network for 12 hours. During this time, a new index is rolled. When the network comes online, filebeat/logstash will catch up and index the older logs. Will these older logs appear in the new index? If so, it seems that deleting old logs by index does not guarantee that logs will be deleted by @timestamp order anymore. This isn't particularly an issue, but just wondering.
Another question - how does logstash discover the new index name to send index requests to? Does it just index to a pattern and ES handles the correct index?