EDIT: the "\t"'s that appear on this page are all supposed to be double slashes. I can't really figure out how to correct them all on this page but if I'm missing any in these filters its probably a formatting issue instead of the cause for my issues.
I have filebeat running on an hMailServer installation. I'm trying to parse the AWSTATS logs and its throwing me for one loop after another. I can't even get the timestamp out of these logs for some reason. Here's one of my thousand config attempts that all lead to the same following failure. I'm fairly sure my "\t" field separator that works in the debugger may not work at all in my Logstash.
PATTERNS:
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b)
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))(.?|\b)
IPORHOST (?:%{HOSTNAME}|%{IPV4})
input {
beats {
port => 5432
}
stdin {
}
}
filter {
grok {
match => [ 'message', '%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}/\t%{EMAILADDRESS:addr1}/\t%{EMAILADDRESS:addr2}/\t%{IPV4:sender_IP' ]
}
mutate {
add_tag => "awstats"
}
}
output {
stdout {
codec => "rubydebug"
}
}
Here's the exact structure of all the logs in this file:
2017-05-09 01:03:49\tThe_Shepards_Diet@parts.yshepardweightshows.top\talitte@redact.com\t93.79.110.8\t127.0.0.1\tSMTP\t?\t554\t0
The exact message when opened in notepad does not have these \t separators. In the grok debugger they parse just fine using "\t" in between each field. In Logstash I've never gotten a successful output. I have tried countless filters like such:
(some missing the last fields due to frusterated version control but I still expect them to capture the first few fields)
%{TIMESTAMP_ISO8601:timestamp}\t%{EMAILADDRESS:addr1}\t%{EMAILADDRESS:addr2}\t%{IPV4:sender_IP}
%{TIMESTAMP_ISO8601:timestamp}\s+%{EMAILADDRESS:email_from}\s+%{EMAILADDRESS:email_to}\s+%{IPV4:sender_IP}\s+%{IPORHOST:originating_IP}\s+%{IPORHOST:SMTP_server}\s+%{WORD:message_type}\s+%{GREEDYDATA:dropme}\s+%{POSINT:SMTP_code}\s+%{POSINT:session_id}
{
"@timestamp" => 2017-05-09T05:03:52.178Z,
"offset" => 10628361,
"@version" => "1",
"beat" => {
"hostname" => "hMail1",
"name" => "hMail1",
"version" => "5.4.0"
},
"input_type" => "log",
"host" => "hMail1",
"source" => "e:\hMailServer\Logs\hmailserver_awstats.log",
"message" => "2017-05-09 01:03:49\tThe_Shepards_Diet@parts.yshepardweightshows.top\talitte@redact.com\t93.79.110.8\t127.0.0.1\tSMTP\t?\t554\t0",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_grokparsefailure",
[2] "awstats"
]
}