I'm curious about how filebeat behaves on a given path. My goal is to ingest IIS logs as filebeat -> logstash -> elasticsearch. The IIS log directory has log files going back to Jan 2016 (1 per day). I am using 'ignore_older' in my filebeat.yml file to ingest approximately the last 2 months of log information as:
- I was unable to use 'd' as a directive as it said it was 'unknown'. Does ignore_older just accept 'h' or 's'?
- In Kibana, when I filter the index on log_timestamp, the oldest date I see is: 2017-08-13. Can I assume that the ingest of logs has not yet brought in any logs older than that date?
- Is there a better/more efficient way to accomplish what I'm looking to do?
I am using the following in my filebeat.yml file:
paths: - C:\path\to\files\file_pattern* input_type: log ignore_older: 1344h
- Is using the 'file_pattern*' causing any inefficiency? Should I just use \path\to\file\dir if all the files in the directory are specific to what I want to ingest?
- How does filebeat traverse the designated path? Does it start with the oldest file timestamp and work it's way to the newer files (this would seem not to be the case given the above date filtering on log_timestamp)?
- Does filebeat skip files that have an older time stamp than the designated ignore_older directive or does it still dow some sort of parsing?
I am happy to read any documentation I can be pointed to. I appreciate any guidance.