Filebeat showing logs older than 24 hrs

[filebeat_win]

Hi All,

On Windows, my filebeat.yml is set to ignore older than 24h, yet when I am running filebeat today, it is actually showing IIS logs from back in May. The IIS logs are configured to rollover each day, so I'm not sure what is causing this. Any thoughts?

To further clarify, Kibana only shows entries for 5/26, although I just opened the IIS website from the server in question moments ago. A new IIS log file was created today, and several other IIS log files exist from different days, dating back to 2014. I am unsure why it chose to display log files from 5/24 only.

[filebeat_win]

Screenshot attached of the IIS log directory, highlighting the file that some logs are pulling from. Note that not all log entries within the log file are showing in Kibana, only select ones. I do not see any logs pulled from previous days, or anything from today's log.

This is running filebeat 1.2.3, and IIS has been restarted.

[ruflin]

I'm not sure I can follow your description to 100%. Here is what I get:

• You have ignore_older set to 24h
• In Kibana it shows correctly only the most recent files after harvesting?
• Filebeat crawls more files (how did you spot that if they are not in Kibana)?

Can you share your config file?

[filebeat_win]

So as of now (2:00 PM UTC), I the newest logs that I see are from 2016-06-14 at 00:00:00. Strangely when I sort by log_timestamp they are out of order.

The filebeat config:

filebeat:
prospectors:
paths:
- C:\inetpub\logs\LogFiles**
- D:\logfiles**
encoding: utf-8
exclude_lines: ["^#"]
exclude_files: [".zip"]
ignore_older: 24h
registry_file: "C:/ProgramData/filebeat/registry"
output:
logstash:
hosts: ["elk:3525"]
compression_level: 0
logging:
files:
rotateeverybytes: 10485760 # = 10MB

[ruflin]

Are these two posts related / the same? Filebeat Windows Config Help (filebeat.yml)

[filebeat_win]

Yes - they are related. I will comment on the other one with more information.