SMTP logs from Windows Server can't be parsed

[Vadym_Mykolaichuk]

Hi!

I'm using filebeat to send IIS log files from Windows Server machine. It parses logs from IIS (web logs) just fine, but it can't parse SMTP logs which have the same format.
On Kibana I see this error:
Provided Grok expressions do not match field value:
[2020-04-27 06:43:55 127.0.0.1 hostname SMTPSVC1 hostname 127.0.0.1 0 EHLO - +hostname 250 0 185 16 0 SMTP - - - -]

Please advise what I supposed to do to see parsed SMTP logs. Let me know if any extra info is needed.

[Kerry (Kerry Gallagher)]

Hi @Vadym_Mykolaichuk,

Sorry to hear you're having problems. Could you please provide the following information:

• Are you using Filebeat "as-is" (using the Filebeat index templates, using the IIS module etc)? Or do you have something like an Ingest pipeline in between?

• What is the Grok expression that you're using?

• What version of Filebeat are you using?

[Vadym_Mykolaichuk]

Hi @Kerry!
I'm using IIS module for filebeat 7.6.1 version "as-is" with default grok patterns.
Here are these patterns from this file:
C:\Program Files\Elastic\Beats\7.6.1\filebeat\module\iis\access\ingest
default.json

"%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:destination.address} %{WORD:http.request.method} %{NOTSPACE:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NOTSPACE:http.request.referrer} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:temp.duration:long}",

"%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{WORD:http.request.method} %{NOTSPACE:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:http.request.referrer} %{NOTSPACE:destination.domain} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:http.response.body.bytes:long} %{NUMBER:http.request.body.bytes:long} %{NUMBER:temp.duration:long}",

"%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{NOTSPACE:iis.access.server_name} %{IPORHOST:destination.address} %{WORD:http.request.method} %{NOTSPACE:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} %{IPORHOST:source.address} HTTP/%{NUMBER:http.version} %{NOTSPACE:user_agent.original} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:http.request.referrer} %{NOTSPACE:destination.domain} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:http.response.body.bytes:long} %{NUMBER:http.request.body.bytes:long} %{NUMBER:temp.duration:long}",

"%{TIMESTAMP_ISO8601:iis.access.time} \\[%{IPORHOST:destination.address}\\]\\(http://%{IPORHOST:destination.address}\\) %{WORD:http.request.method} %{NOTSPACE:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} \\[%{IPORHOST:source.address}\\]\\(http://%{IPORHOST:source.address}\\) %{NOTSPACE:user_agent.original} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:temp.duration:long}",

"%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:destination.address} %{WORD:http.request.method} %{NOTSPACE:url.path} %{NOTSPACE:url.query} %{NUMBER:destination.port:long} %{NOTSPACE:user.name} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NUMBER:http.response.status_code:long} %{NUMBER:iis.access.sub_status:long} %{NUMBER:iis.access.win32_status:long} %{NUMBER:temp.duration:long}"

Here are some example of logs (data has been modified):

#Software: Microsoft Internet Information Services 10.0
#Version: 1.0
#Date: 2020-04-27 06:43:55
#Fields: date time c-ip cs-username s-sitename s-computername s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status sc-win32-status sc-bytes cs-bytes time-taken cs-version cs-host cs(User-Agent) cs(Cookie) cs(Referer) 
2020-04-27 06:43:55 127.0.0.1 server-01 SMTPSVC1 server-01 127.0.0.1 0 EHLO - +server-01 250 0 185 16 0 SMTP - - - -
2020-04-27 06:43:55 127.0.0.1 server-01 SMTPSVC1 server-01 127.0.0.1 0 MAIL - +FROM:<test@example.com> 250 0 36 23 0 SMTP - - - -
2020-04-27 06:43:55 127.0.0.1 server-01 SMTPSVC1 server-01 127.0.0.1 0 RCPT - +TO:<recipient@example.com> 250 0 42 39 15 SMTP - - - -
2020-04-27 06:43:55 127.0.0.1 server-01 SMTPSVC1 server-01 127.0.0.1 0 DATA - <server-01YRofCD2h00000001@server-01> 250 0 124 291 16 SMTP - - - -
2020-04-27 06:43:55 8.8.8.8 OutboundConnectionResponse SMTPSVC1 server-01 - 25 - - 220+MS+ESMTP+service+ready+at+ismtpd0002p1lon1.mailservice.com 0 0 59 0 15 SMTP - - - -
2020-04-27 06:43:55 8.8.8.8 OutboundConnectionCommand SMTPSVC1 server-01 - 25 EHLO - server-01 0 0 4 0 15 SMTP - - - -
2020-04-27 06:43:55 8.8.8.8 OutboundConnectionResponse SMTPSVC1 server-01 - 25 - - 250-smtp.mailservice.com 0 0 21 0 31 SMTP - - - -
2020-04-27 06:43:55 8.8.8.8 OutboundConnectionCommand SMTPSVC1 server-01 - 25 AUTH - - 0 0 4 0 31 SMTP - - - -

[Kerry (Kerry Gallagher)]

Great, thank you.

Those Grok patterns are definitely incompatible with the log examples you've provided. It looks like you'll need to add some extra patterns to cover the SMTP scenario.

We have a Grok debugger available at app/kibana#/dev_tools/grokdebugger which you can use to help with the creation of patterns.

Using the fields listed for the IIS module you'll be able to create a suitable pattern that follows
date time c-ip cs-username s-sitename s-computername s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status sc-win32-status sc-bytes cs-bytes time-taken cs-version cs-host cs(User-Agent) cs(Cookie) cs(Referer)

Something like this should get you started:

%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:iis.access.remote_ip} %{NOTSPACE:iis.access.user_name} %{NOTSPACE:iis.access.site_name} %{NOTSPACE:iis.access.server_name} %{IPORHOST:iis.access.server_ip} %{INT:iis.access.port}

Hopefully that helps. If you need further help the Beats board will be able to advise you further on the Filebeat configuration.