Filter Multiple Patterns

[Bhuvanesh]

Hi All,

I have an ELK integrated log server. Normally I get all the messages in the format

%{SYSLOGTIMESTAMP} %{SYSLOGHOST:sysloghost} %{SYSLOGPROG:syslogprog}: %{GREEDYDATA:Message}

I have two level filters where I check eg : if syslogprog == "apache-access" then do further filtering of the messages. This is working fine.

But, now I have added a new log where the format has a small difference,

%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:sysloghost} %{USER:systemuser}: %{SYSLOGPROG:syslogprog} %{GREEDYDATA:Program}

The log I am referring in the format : -

Jul 31 23:39:17 Server-3 reyan: User-Activity root 100.10.56.1 [7372]: 31 Jul 23:39 df -h

See, here the syslogprog field is shifted to another position. Whenever I am adding this new filter configuration logstash throws error in its log.

How to deal with this case ? Please advise.

Regards,
Bhuvanesh

[magnusbaeck (Magnus Bäck)]

Grok filters supports multiple expressions (there's an example in the docs). The expressions will be tried one by one in the order you list them. I think it'll work for you if you list the longer expression first so that Logstash only falls back to the more generic expression if the first one doesn't match.

See, here the syslogprog field is shifted to another position. Whenever I am adding this new filter configuration logstash throws error in its log.

Always tell us exactly what you tried and exactly what error you get.

[Bhuvanesh]

Hi Magnus,

Thanks for the reply.

My two filter files are as follows :-

File name : 03-apachef.conf

filter {
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP} %{SYSLOGHOST:sysloghost} %{SYSLOGPROG:syslogprog}: "
}
}
if [syslogprog] == "apache-access" {

grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:sysloghost} %{SYSLOGPROG:syslogprog}: %{COMBINEDAPACHELOG}" }
}

date {
  match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  remove_field => [ "timestamp", "message", "path", "ident", "auth", "logsource" ]
}

}

}

File name : 04-activity.conf

filter {
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP} %{SYSLOGHOST:sysloghost} %{SYSLOGUSER:systemuser}: %{SYSLOGPROG:syslogprog} "
}
}

if [syslogprog] == "User-Activity" {

grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:sysloghost} %{USER:systemuser}: %{SYSLOGPROG:syslogprog} %{USER:escalatedto} %{IP:sourceip} [%{NUMBER:Number}]: %{GREEDYDATA:Program}" }
}

}
}

First filter works fine, But when I add the second filter I get an error like this

[2017-08-25T01:57:35,869][ERROR][logstash.agent] Pipeline aborted due to error {:exception=>#<Grok::PatternError: pattern %{SYSLOGUSER:systemuser} not defined>

Please advise.

Thanks,
Bhuvanesh

[magnusbaeck (Magnus Bäck)]

Your Logstash and its plugins doesn't support a SYSLOGUSER pattern. Maybe that pattern was introduced in later versions. You can use another pattern instead, e.g. WORD or NOTSPACE.