Grok Parse failure with fail2ban

Elastic Stack Logstash
1

I followed 2 different guides and I still havent figure out where the issue is. I want to parse fail2ban logs and I tried the following two configurations unsuccessfully!

Pattern files that I tried

root@leeds:/etc/logstash/patterns# cat /etc/logstash/patterns/fail2ban
F2B_DATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{HOUR}:?%{MINUTE}(?::?%{SECOND})
F2B_ACTION (\w+).(?:\w+)(\s+)?:
F2B_JAIL [(?\w+-?\w+?)]
F2B_LEVEL (?\w+)\s+

and

root@leeds:/etc/logstash/patterns# cat /tmp/fail2ban2
FAIL2BAN_BAN %{TIMESTAMP_ISO8601:timestamp} %{JAVACLASS:criteria}: %{LOGLEVEL:level} [%{WORD:service}] Ban %{IPV4:clientip}
FAIL2BAN_UNBAN %{TIMESTAMP_ISO8601:timestamp} %{JAVACLASS:criteria}: %{LOGLEVEL:level} [%{WORD:service}] Unban %{IPV4:clientip}
FAIL2BAN_ALREADYBAN %{TIMESTAMP_ISO8601:timestamp} %{JAVACLASS:criteria}: %{LOGLEVEL:level} [%{WORD:service}] %{IPV4:clientip} already banned

Logstash confs

file {
type => "fail2ban"
path => "/tmp/fail2ban.log"
start_position => "beginning"
document_type => fail2ban
}
filter {
if [type] == "fail2ban" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => [
"message", "%{F2B_DATE:date} %{F2B_ACTION} %{WORD:level} %{F2B_JAIL} %{WORD:action} %{IP:ip}",
"message", "%{F2B_DATE:date} %{F2B_ACTION} %{F2B_LEVEL} %{GREEDYDATA:msg}?"
]
}
}
}

Output is fine so I am not listing it here

For the second pattern test the relevant part is here

if [type] == "fail2ban" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => [ "message", "%{FAIL2BAN_BAN}" ]
}
}

The permissions of the /etc/logstash/pattern are set with chown to logstash user and the above filters lead to a grokparsefailure.

The log looks like this...

2016-01-26 05:12:20,778 fail2ban.server [1148]: INFO rollover performed on /var/log/fail2ban.log
2016-01-26 05:12:20,828 fail2ban.actions [1148]: NOTICE [bruteforce3] Ban 255.255.255.255
2016-01-26 05:12:21,048 fail2ban.actions [1148]: NOTICE [bruteforce3] Ban 255.255.255.255
2016-01-26 05:12:21,064 fail2ban.filter [1148]: INFO Log rotation detected for /var/log/fail2ban.log
2016-01-26 05:12:21,064 fail2ban.filter [1148]: INFO Log rotation detected for /var/log/fail2ban.log
2016-01-26 05:12:21,065 fail2ban.filter [1148]: INFO Log rotation detected for /var/log/fail2ban.log
2016-01-26 05:12:21,266 fail2ban.actions [1148]: NOTICE [bruteforce3] Ban 255.255.255.255

Any help?

2

It doesn't look like you're taking the pid (or whatever it is) into account, i.e. [1148] in your example.

3

Is there a way to skip it? Is this why im getting the grok failure?

4

You don't have to keep the pid in a field but you have to include it in your expression, e.g. with \[%{INT}\].

5

Didnt work :confused:

6

Then you have to debug more systematically. Start with the simplest possible expression, %{TIMESTAMP_ISO8601:timestamp}. Does it work? If yes, add the next part of the expression. Continue until things stop working.

7

You can use the grok debugger tool...

https://grokdebug.herokuapp.com/

8

%{TIMESTAMP_ISO8601:timestamp} %{JAVACLASS:criteria} [%{NUMBER:pid}]: %{LOGLEVEL:level} [%{NOTSPACE:service}] Found %{IPV4:clientip}

Explanation:
1)Take attention to the spaces...
2)No match for pid was present (That's where the 2 points go.)
3)Service is not a word (Word is alphanumerical and underscores). Here is a dash, so we use NOTSPACE

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.