Logstash Multiple Grok Add Field

Elastic Stack Logstash
1

I'm trying to have a single message parse through a hash array of match, and then check the message type after to add a field.

 grok {
  match => { "message" => [
      "%{NOTSPACE:syslogversion}%{SYSLOGTIMESTAMP:timestamp} %{WORD:servicename}\[%{NUMBER:pid}\]: Invalid user %{WORD:username} from %{IP:ip}",
      "%{NOTSPACE:syslogversion}%{SYSLOGTIMESTAMP:timestamp} %{WORD:servicename}\[%{NUMBER:pid}\]: Failed keyboard-interactive/pam for invalid user %{WORD:username} from %{IP:ip} port %{NUMBER:port} %{WORD:protocol}"
      "%{NOTSPACE:syslogversion}%{SYSLOGTIMESTAMP:timestamp} %{WORD:servicename}\[%{NUMBER:pid}\]: Accepted keyboard-interactive/pam for %{WORD:username} from %{IP:ip} port %{NUMBER:port} %{WORD:protocol}",
      "%{NOTSPACE:syslogversion}%{SYSLOGTIMESTAMP:timestamp} %{WORD:servicename}\[%{NUMBER:pid}\]: Failed password for %{WORD:username} from %{IP:ip} port %{NUMBER:port} %{WORD:protocol}",
      "%{NOTSPACE:syslogversion}%{SYSLOGTIMESTAMP:timestamp} %{DATA:servicename}\[%{NUMBER:pid}\]: /index.php: Successful login for user '%{WORD:username}' from: %{IP:ip}",
      "%{NOTSPACE:syslogversion}%{SYSLOGTIMESTAMP:timestamp} %{DATA:servicename}\[%{NUMBER:pid}\]: /index.php: User logged out for user '%{WORD:username}' from: %{IP:ip}",
      "%{NOTSPACE:syslogversion}%{SYSLOGTIMESTAMP:timestamp} %{DATA:servicename}\[%{NUMBER:pid}\]: /index.php: webConfigurator authentication error for '%{WORD:username}' from %{IP:ip}"
      ]
  }
}

    if "Failed keyboard" in [message] {
      grok { add_field => { "error_type" => "Failed keyboard-interactive/pam" } }
    }

    if "Invalid User" in [message] {
      grok { add_field => { "error_type" => "Invalid User" } }
    }


    if "Accepted keyboard" in [message] {
      grok { add_field => { "error_type" => "Accepted User" } }
    }

    if "Failed password" in [message] {
      grok { add_field => { "error_type" => "Invalid Password" } }
    }

    if "Successful login" in [message] {
      grok { add_field => { "error_type" => "Successful Login" } }
    }

    if "User logged out" in [message] {
      grok { add_field => { "error_type" => "Logged Out" } }
    }

    if "webConfigurator authentication error" in [message] {
      grok { add_field => { "error_type" => "Invalid Password" } }
    }

However, the message does get grok'd fine, but its always has the _grokparsefailure tag, and the additional field does not get added.

{"ip":"10.162.126.165","pid":"11609","username":"test","tags":["_grokparsefailure"],"servicename":"php-fpm","message":"<32>Jul 26 10:18:15 php-fpm[11609]: /index.php: webConfigurator authentication error for 'test' from 10.162.126.165","@version":"1","@timestamp":"2018-07-26T02:18:15.000Z","type":"pfsense-system","timestamp":"Jul 26 10:18:15","syslogversion":"<32>"}

Can anyone point me in the right direction? All the threads i've found haven't helped.

2

Don't use a grok filter to just add a field. Use a mutate filter for that. add_field only triggers when the filter is successful, and a grok filter without match doesn't count as successful.

3

Thank you Magnus. Two birds with one stone :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.