_grokparsefailure, _geoip_lookup_failure sophos utm

[fooc (sander van der Laan)]

Hi,

I'm using the following conf file to process my syslog messages from a sophos utm:

input {
tcp {

type => "utm-syslog"

port => 5140

}
}

filter {

if [type] == "utm-syslog" {

grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{DATA:syslog_program}[%{NUMBER:syslog_pid}]: %{GREEDYDATA:syslog_message}" }
}
date {
match => [ "syslog_timestamp", "yyyy:MM:dd-HH:mm:ss" ]
}
kv {
source => "syslog_message"
}
mutate {
replace => [ "type", "%{syslog_program}" ]
remove_field => [ "syslog_message", "syslog_timestamp" ]
}

if [type] == "httpproxy" {
grok { match => { "url" => "(?https?)://%{IPORHOST:url_domain}/" } }

}
geoip {
source => "dstip"
}
geoip {
source => "srcip"
}

} # end of filter

output {

if [type] == "utm-syslog" {

elasticsearch {
hosts => ["10.255.254.27:9200"]
index => "logstash-utm-%{+YYYY.MM.dd}"

}

stdout { codec => rubydebug }

}

In normal firewall messages the geoip works well, but in messages from the waf (web application firewall) i get the grokparse failure messages:

In the message a srcip is present, i can't understand why I'm getting the errors.

[Badger]

You grok pattern expects syslog_program to be followed by syslog_pid in square brackets. Your message does not have that so it does not match.

Please do not post screenshots of text, just post the text.

[fooc (sander van der Laan)]

@Badger

square brackets

Can I just remove the square brackets?

[Badger]

Removing the square brackets is an option, but I assume they were added to the pattern because you have some lines that contain them. In that case you can match against multiple patterns using

grok { match { "message" => [ "firstPattern", "secondPattern" ] } }

Where one pattern has the brackets and one does not.

BTW you should anchor your grok patterns whenever possible.

match => { "message" => "<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{DATA:syslog_program}[%{NUMBER:syslog_pid}]: %{GREEDYDATA:syslog_message}" }

is a lot slower than

match => { "message" => "^<%{POSINT:syslog_pri}>%{DATA:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{DATA:syslog_program}[%{NUMBER:syslog_pid}]: %{GREEDYDATA:syslog_message}" }

[fooc (sander van der Laan)]

Thanx for your quick reply, i copied the conf from internet and the basics were working for me, just not the waf messages.
I have to learn how to write it.