[Solved] Logstash pattern issue

Elastic Stack Logstash
1

Hi,

I came across a weird issue.

I made a grok pattern that works well and matches my log in http://grokdebug.herokuapp.com, but when I use the same pattern in my logstash, its not working properly. I get a field mismatch. Please check below:

This is the log:

Tue Mar 7 12:23:41 2017 : Auth: (49087) Login incorrect (eap_peap: TLS Alert write:fatal:decode error): [johndoe] (from client WIFI-Control-Dev port 13 cli bb-41-e2-1c-12-12)

Grok pattern:

(%{NOTSPACE:removed}) %{DATA:AUTHWAY} (?:(%{DATA:removed}))?: (?:[%{DATA:username}]) (from client %{NOTSPACE:radius_client} port %{INT:radius_port}(?: cli %{NOTSPACE:radius_cli})?(?: via %{DATA:radius_via})?)

In Grok debugger, its working but not in my server. In my server, the logstash config read as:

filter {
if [type] == "radius-auth" {
grok {
patterns_dir => "/etc/logstash/patterns/"
match => [ "message", "%{FREERADIUS_LINE}" ]
}
if [fr_logclass] == "Auth" {
grok {
patterns_dir => "/etc/logstash/patterns/"
match => [ "fr_message", "%{FREERADIUS_AUTH}" ]
}
}
date {
match => [ "timestamp" , "EEE MMM dd HH:mm:ss YYYY",
"EEE MMM d HH:mm:ss YYYY"]
}
}

/etc/logstash/patterns/radius-auth.pattern contains

FREERADIUS_DATE %{DAY} %{MONTH}  ?%{MONTHDAY} %{TIME} %{YEAR}
FREERADIUS_LOGTYPE Auth|Info|Error|Proxy
FREERADIUS_CLIENT %{NOTSPACE:radius_client}
FREERADIUS_PORT %{INT:radius_port}
FREERADIUS_CLI %{NOTSPACE:radius_cli}
FREERADIUS_VIA %{DATA:radius_via}
FREERADIUS_FROM \(from client %{FREERADIUS_CLIENT} port %{FREERADIUS_PORT}(?: cli %{FREERADIUS_CLI})?(?: via %{FREERADIUS_VIA})?\)
FREERADIUS_USERNAME %{DATA:username}
FREERADIUS_REASON (?:\(%{DATA:radius_reason}\))?:
FREERADIUS_MODULE \brlm_[a-z]+\b

# Auth log lines
FREERADIUS_LOGINOK \(%{NOTSPACE:removed}\) %{GREEDYDATA:AUTHWAY}: \[%{FREERADIUS_USERNAME}\] %{FREERADIUS_FROM}
FREERADIUS_LOGININCORRECT \(%{NOTSPACE:removed}\) %{DATA:AUTHWAY} (?:\(%{DATA:radius_reason}\))?: (?:\[%{DATA:username}\]) \(from client %{NOTSPACE:radius_client} port %{INT:radius_port}(?: cli %{NOTSPACE:radius_cli})?(?: via %{DATA:radius_via})?\)
FREERADIUS_INVALIDUSER \(%{NOTSPACE:removed}\) %{GREEDYDATA:AUTHWAY}: \[%{FREERADIUS_USERNAME}\] %{FREERADIUS_FROM}
FREERADIUS_AUTH (?:%{FREERADIUS_LOGINOK}|%{FREERADIUS_LOGININCORRECT}|%{FREERADIUS_INVALIDUSER})

# Main match on whole log line:
FREERADIUS_LINE %{FREERADIUS_DATE:timestamp} : %{FREERADIUS_LOGTYPE:fr_logclass}: +%{GREEDYDATA:fr_message}

But in my server, the radius_reason field is not getting separated and its merging with AUTHWAY field.

Ruby debug:

{
       "fr_message" => "(49087) Login incorrect (eap_peap: TLS Alert write:fatal:decode error): [johndoe] (from client WIFI-Control-Dev port 13 cli bb-41-e2-1c-12-12)",
           "offset" => 185,
          "AUTHWAY" => "Login incorrect (eap_peap: TLS Alert write:fatal:decode error)",
       "input_type" => "log",
      "radius_port" => "13",
           "source" => "/var/log/radius/radius.log",
          "message" => "Tue Mar  7 12:23:41 2017 : Auth: (49087) Login incorrect (eap_peap: TLS Alert write:fatal:decode error): [johndoe] (from client WIFI-Control-Dev port 13 cli bb-41-e2-1c-12-12)",
             "type" => "radius-auth",
             "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
       "@timestamp" => 2017-03-07T12:23:41.000Z,
      "fr_logclass" => "Auth",
          "removed" => "49087",
    "radius_client" => "WIFI-Control-Dev",
         "@version" => "1",
             "beat" => {
        "hostname" => "localhost.localdomain",
            "name" => "localhost.localdomain",
         "version" => "5.1.1"
    },
             "host" => "localhost.localdomain",
       "radius_cli" => "bb-41-e2-1c-12-12",
        "timestamp" => "Tue Mar  7 12:23:41 2017",
         "username" => "johndoe"
}

The AUTHWAY field should only contain "Login Incorrect" . But it is also accompanied by "(eap_peap: TLS Alert write:fatal:decode error)" which should be in separate field as "radius_reason". In online grok debugger, both displayed as separate field, but not in my server.

Any idea how to resolve?

Thank you.

2

I don't have time to dig into all the details, but I strongly suggest that you reduce your use of GREEDYDATA and DATA, especially since you have optional matches (the ? operator). Try to make more exact expressions.

3

Ok, thank you . I will check further. I'm with this issue since 5 days :wink: and couldn't figure out why.

4

So after thinking further, I found a way to make it work.

This is the grok pattern I used and it works both in grok debugger and in server.

(%{DATA:removed}) %{DATA:AUTHWAY}(?::)? (?:((.*?)):)?%{SPACE}[%{DATA:username}] (from client %{NOTSPACE:radius_client} port %{INT:radius_port}(?: cli %{NOTSPACE:radius_cli})?(?: via %{DATA:radius_via})?)

Here I used (?::)? (?:\((.*?)\):)?%{SPACE} instead of (?:\(%{DATA:radius_reason}\))?: which was in the grok pattern earlier I posted in first post.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.