I have filebeat installed with the iis module, and it is using the default iis error pipeline for the HTTPERR log files.
A few of the error logs have malformed urls, which the grok patterns don't match.
here's the grok pattern for the filebeat-6.3.0-iis-error-default pipeline:
%{TIMESTAMP_ISO8601:iis.error.time} %{IPORHOST:iis.error.remote_ip} %{NUMBER:iis.error.remote_port} %{IPORHOST:iis.error.server_ip} %{IPORHOST:iis.error.server_port} (?:HTTP/%{NUMBER:iis.error.http_version}|-) (?:%{WORD:iis.error.method}|-) (?:%{URIPATHPARAM:iis.error.url}|-)(?: -)? (?:%{NUMBER:iis.error.response_code}|-) (?:%{NUMBER}|-) (?:%{NOTSPACE:iis.error.reason_phrase}|-) (?:%{NOTSPACE:iis.error.queue_name}|-)
And here are a few examples of the format of the logs that grok failed to parse
2018-05-05 05:05:55 149.42.83.135 12345 192.168.101.101 443 HTTP/0.9 t3 12.2.1 400 - URL -
2018-05-05 05:05:55 149.42.83.135 12345 192.168.101.101 443 HTTP/1.1 GET ./././././../../../../../../../../ 400 - URL -
2018-05-05 05:05:55 149.42.83.135 12345 192.168.101.101 443 HTTP/1.1 GET /..\pixfir~1\how_to_login.html 403 - Forbidden -
2018-05-05 05:05:55 149.42.83.135 12345 192.168.101.101 443 HTTP/1.1 GET ..\..\..\..\..\..\winnt\win.ini 400 - URL -
2018-05-05 05:05:55 149.42.83.135 12345 192.168.101.101 443 HTTP/1.1 GET /�.�./�.�./�.�./�.�./�.�./windows/win.ini 404 - NotFound -
2018-05-05 05:05:55 149.42.83.135 12345 192.168.101.101 443 HTTP/1.1 GET /nessus\..\..\..\..\..\..\winnt\win.ini 403 - Forbidden -
2018-05-05 05:05:55 149.42.83.135 12345 192.168.101.101 443 HTTP/1.1 OPTIONS * 404 - NotFound -
2018-05-05 05:05:55 149.42.83.135 12345 192.168.101.101 443 HTTP/1.1 GET /fee&fie=foe 400 - URL -
Since the grok parser fails at the start of the pipeline, none of the fields get populated, so these logs are not stored properly.
I'm wondering if the part of the grok pattern that looks at the uri should be changed from (?:%{URIPATHPARAM:iis.error.url}|-)to something that captures everything that isn't a space or -?
Maybe (?:(?<iis.error.url>[^\s\-]+)|-)
Thanks