Grok Parse failure with fail2ban

I followed 2 different guides and I still havent figure out where the issue is. I want to parse fail2ban logs and I tried the following two configurations unsuccessfully!

Pattern files that I tried

root@leeds:/etc/logstash/patterns# cat /etc/logstash/patterns/fail2ban
F2B_ACTION (\w+).(?:\w+)(\s+)?:
F2B_JAIL [(?\w+-?\w+?)]
F2B_LEVEL (?\w+)\s+


root@leeds:/etc/logstash/patterns# cat /tmp/fail2ban2
FAIL2BAN_BAN %{TIMESTAMP_ISO8601:timestamp} %{JAVACLASS:criteria}: %{LOGLEVEL:level} [%{WORD:service}] Ban %{IPV4:clientip}
FAIL2BAN_UNBAN %{TIMESTAMP_ISO8601:timestamp} %{JAVACLASS:criteria}: %{LOGLEVEL:level} [%{WORD:service}] Unban %{IPV4:clientip}
FAIL2BAN_ALREADYBAN %{TIMESTAMP_ISO8601:timestamp} %{JAVACLASS:criteria}: %{LOGLEVEL:level} [%{WORD:service}] %{IPV4:clientip} already banned

Logstash confs

file {
type => "fail2ban"
path => "/tmp/fail2ban.log"
start_position => "beginning"
document_type => fail2ban
filter {
if [type] == "fail2ban" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => [
"message", "%{F2B_DATE:date} %{F2B_ACTION} %{WORD:level} %{F2B_JAIL} %{WORD:action} %{IP:ip}",
"message", "%{F2B_DATE:date} %{F2B_ACTION} %{F2B_LEVEL} %{GREEDYDATA:msg}?"

Output is fine so I am not listing it here

For the second pattern test the relevant part is here

if [type] == "fail2ban" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => [ "message", "%{FAIL2BAN_BAN}" ]

The permissions of the /etc/logstash/pattern are set with chown to logstash user and the above filters lead to a grokparsefailure.

The log looks like this...

2016-01-26 05:12:20,778 fail2ban.server [1148]: INFO rollover performed on /var/log/fail2ban.log
2016-01-26 05:12:20,828 fail2ban.actions [1148]: NOTICE [bruteforce3] Ban
2016-01-26 05:12:21,048 fail2ban.actions [1148]: NOTICE [bruteforce3] Ban
2016-01-26 05:12:21,064 fail2ban.filter [1148]: INFO Log rotation detected for /var/log/fail2ban.log
2016-01-26 05:12:21,064 fail2ban.filter [1148]: INFO Log rotation detected for /var/log/fail2ban.log
2016-01-26 05:12:21,065 fail2ban.filter [1148]: INFO Log rotation detected for /var/log/fail2ban.log
2016-01-26 05:12:21,266 fail2ban.actions [1148]: NOTICE [bruteforce3] Ban

Any help?

It doesn't look like you're taking the pid (or whatever it is) into account, i.e. [1148] in your example.

Is there a way to skip it? Is this why im getting the grok failure?

You don't have to keep the pid in a field but you have to include it in your expression, e.g. with \[%{INT}\].

Didnt work :confused:

Then you have to debug more systematically. Start with the simplest possible expression, %{TIMESTAMP_ISO8601:timestamp}. Does it work? If yes, add the next part of the expression. Continue until things stop working.

You can use the grok debugger tool...

%{TIMESTAMP_ISO8601:timestamp} %{JAVACLASS:criteria} [%{NUMBER:pid}]: %{LOGLEVEL:level} [%{NOTSPACE:service}] Found %{IPV4:clientip}

1)Take attention to the spaces...
2)No match for pid was present (That's where the 2 points go.)
3)Service is not a word (Word is alphanumerical and underscores). Here is a dash, so we use NOTSPACE

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.