Data Accuracy with Filebeat/IIS Aggregation

[subhashpant]

I was previously parsing the IIS Log data (to get aggregate) using Log Parser 2.2. Generic command as follows:

.\logparser -rtp:-1 -i:iisw3c "SELECT TO_LOCALTIME(QUANTIZE(TO_TIMESTAMP(date, time), 3600)) AS [DateTime], COUNT(*) AS [Requests] INTO HourlyDayReport_OCSP01.txt FROM C:\DBATools\Log\u_ex190731.log GROUP BY [DateTime] ORDER BY [DateTime]"

I installed 7.3 version of ELK stack, and using Filebeat 7.3 also. No customization, just enabled IIS module via Filebeat and data is getting ingested. The IIS Log is in W3C format, so the time is in GMT. The ingestion pipeline is IIS Logs -> FileBeat 3.0 -> Elastic Search 7.3. No logstash or custom parsing.

Issue:

1. During some hours, aggregates do not match by a small quantity (This did not pique my attention much, as, there could be diff between how Log Parser vs Elastic Search interprets the time). However, they are very close (interpreted in same hour - CST).
2. There are several hours on each day where aggregates (count of hits) are < 500, but the logs have those data.

Also uploaded the raw data into Azure Data Analytics and built the same aggregation using U-SQL. This matches the result of Log Parser.

I looked into errors, nothing in ELK stack or syslog (for Kibana). It did not make sense to me why there would be a gap (i.e. if it stopped working, there would be no data beyond that point).