How to make make an if condition with a regular expression pattern

Elastic Stack Logstash
1

Hi,

i'm new to ELK and trying to filter some spam lines from being added to the elasticsearch server.
to do this i made the following logstash filter:

## var_log_messages filter
filter {
        if [type] == "var_log_messages" {
                grok {
                        match => { "message" => "%{SYSLOGTIMESTAMP:var_log_secure_timestamp} %{SYSLOGHOST:var_log_secure_hostname} %{DATA:syslog_program}(?:\[%{POSINT:var_log_secure_pid}\])?: %{GREEDYDATA:message}" }
                        overwrite => [ "message"]
                }
                        ##ignore spam line
                        if ([message] =~ "Got message type[=]signal sender[=](\:1\.0|org\.freedesktop\.DBus) destination[=]n\/a object[=][/]org[/]freedesktop[/]"){
                            drop{}
                        }
                        else if ([message] =~ "GSSAPI client step ([0-9]{1,2})"){
                            drop{}
                        }
}
}

I want to make the rules more universal so i drops more unwantend log lines like:

so i made the following line but is does not catch all the unwanted lines:
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 object=/org/freedesktop/systemd1/unit/session_2d528_2escope interface=org.freedesktop.DBus.Properties member=Get cookie=9163 reply_cookie=0 error=n/a
Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/login1 interface=org.freedesktop.login1.Manager member=SessionNew cookie=9131 reply_cookie=0 error=n/a
Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=121996 reply_cookie=0 error=n/a
Got message type=signal sender=org.freedesktop.DBus destination=n/a object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameOwnerChanged cookie=2019 reply_cookie=0 error=n/a
Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/user_2d0_2eslice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=180590 reply_cookie=0 error=n/a
Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=UnitRemoved cookie=180633 reply_cookie=0 error=n/a
Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/dev_2dmapper_2dsystemvg_5cx2dtmp_2edevice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=180638 reply_cookie=0 error=n/a
Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/dev_2dmapper_2dsystemvg_5cx2dvar_2edevice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=180647 reply_cookie=0 error=n/a
Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=UnitRemoved cookie=180622 reply_cookie=0 error=n/a
Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/dev_2dmapper_2dsystemvg_5cx2dvar_5flog_5faudit_2edevice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=180565 reply_cookie=0 error=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=GetConnectionUnixUser cookie=5007 reply_cookie=0 error=n/a

(Got|Sent) message type[=](method\_return|method\_call|signal) (sender)\=(n\/a|\:1\.0|org\.freedesktop\.*?([a-zA_Z])) [destination]

the *?([a-zA_Z])) [destination] part is because i'm trying some things

When i test the lines it does not match all of them https://grokconstructor.appspot.com/do/match#result

because within a if condition you can't use a grok patter i'm using regular expression.
my question how can I make an regular expression to match parts of the line ignore som chars en again match chars and ignore some chars.

like:
Sent message type=method_call sender=n/a destination=org.freedesktop> .(ignore some chars here )object=/org/freedesktop/> ignore some chars interface=org.freedesktop.DBus.Properties member=Get cookie=9163 reply_cookie=0 error=n/a

2

i've made my own solution:

by using allowing the following chars 1 or more times [a-zA-Z0-9]+ and [a-zA-Z0-9\/\_]+
the things left we will ignore: member=Get cookie=<number> reply_cookie=0 error=n/a

(Got|Sent) message type[=](method\_return|method\_call|signal) (sender)\=(n\/a|\:1\.0|org\.freedesktop\.[a-zA-Z0-9]+) destination[=](n\/a|\:1\.0|org\.freedesktop\.[a-zA-Z0-9]+) object\=\/org\/freedesktop\/[a-zA-Z0-9\/\_]+ interface\=org\.freedesktop\.[a-zA-Z0-9]+

When testing all rules are matched! :grinning:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.