So, we installed filebeat v7.10.1 and enables the IIS System module in a pretty standard way.
Under heavy load, we get a batch of IIS access logs nicely parsed into the IIS fields: method, user_agents, geolocation, etc. All entries are timestamped at the "000" msec so the @timestamps for all these good messages read like: "Feb 3, 2021 @ 22:24:45.000".
It seems that filebeat then may send another batch of messages after "000" msec. And in this second batch of messages, every single message fails to parse the IIS fields, and we just have the famous:
error.message: Provided Grok expressions do not match field value: [2021-02-02 00:37:45 192.168.112.207 GET /.....
No IIS fields come through in any messages that come in AFTER the first batch that always sends at "000" msec. The first batch of IIS logs that appears at the very beginning of each second is nicely parsed.
When the second of time passes, I then see a small batch of good messages come in with a @timestamp at the "000" msec mark of the next second, then a bunch of bad messages after that first group of good messages with a @timestamp ending in "000". ONLY IIS messages with a timestamp ending in "000" are good.
Is this happening to anyone else?