Clearpass logs [SOLVED]

Dr_Natas · September 25, 2018, 9:41pm

This is my first grok filter, if any one has any input I'm more than welcome to receive any.

I'm able to parse a good portion of Aruba Clearpass logs but this field I'm unable split it by columns e.g. I would like to split that row into what we see bellow. I'm running Logstash 6.4, and I also included a picture of how the row looks. Bellow is my 1st grok filter.

RADIUS: MSCHAP: AD status:Logon failure (0xc000006d) \nMSCHAP: Authentication failed\nEAP-MSCHAPv2: User authentication failure

RADIUS : MSCHAP
AD status : Login failure (0xc000006d)
MSCHAP: Authentication failed
...

filter {
if "aruba" in [tags] {
    grok {
        match => {
            "message" => "%{MONTH:Month}%{SPACE}%{MONTHDAY:MonthDay}%{SPACE}%{TIME:time}%{SPACE}(?<date>\d{4}-\d{2}-\d{2})%{SPACE} %{DATA:unknown_data} %{DATA:time} %{IPV4:Clearpass_IP} %{SYSLOGHOST:syslog_server} %{NUMBER:log_number} %{INT:unknown_data2} %{INT:unknown_data3} %{GREEDYDATA:kv_pairs}"
        }
    }

    kv {
        source => "kv_pairs"
        field_split => ","
    }

    mutate {
        # Original message has been fully parsed, so remove it.
        remove_field => [ "message" ]
        # Remove kv_pairs to avoid duplicate logs
        remove_field => [ "kv_pairs" ]
    }
}
}

[Log Sample]

Sep 25 14:47:22 2018-09-25 14: 47:22,559 10.30.42.9 syslog-server 1453 1 0 Sep 25 14:51:20 2018-09-25 14: 51:20,615 10.30.42.8 server-name 270369 1 0 Common.Username=john.doe,Common.Service=WLAN_Enterprise,Common.Roles=[Guest],RADIUS.Auth-Source=AD:three.fifty.com,RADIUS.Auth-Method=EAP-PEAP,EAP-MSCHAPv2,Common.System-Posture-Token=UNKNOWN,Common.Enforcement-Profiles=PAN-Update-Node, cci_connect Access Profile,Common.Host-MAC-Address=deadbeefdead,Common.NAS-IP-Address=1.2.3.4,Common.Error-Code=216,Common.Alerts=RADIUS: MSCHAP: AD status:Logon failure (0xc000006d) \nMSCHAP: Authentication failed\nEAP-MSCHAPv2: User authentication failure,Common.Request-Timestamp=2018-09-25 14:50:17-07

I took some of my ideas from his guy.

github.com

njohnsn/ClearPassAndELK/blob/master/10-logstash-syslog.conf

input {

# Listen on TCP/UDP port 5514 for CPPM messages
 tcp {
   port => 5514
   type => syslog
 }

 udp {
   port => 5514
   type => syslog
 }

}

filter {

  # if message contains CPPM message
  if [type] == "syslog" and  [message] =~ "CPPM" {

This file has been truncated. show original

guyboertje · September 26, 2018, 12:27pm

You need to be more specific about the above.

Do you want three separate events with the Common.Alerts field set to each line above, but all the rest of the fields duplicated?

Dr_Natas · September 26, 2018, 3:51pm

Yes, I believe having three separate events would be perfect and would make queering data much better. I think if I'm able to create custom fields would be better. I'm new to this but I hope my mistakes would enlighten the next person.

guyboertje · September 26, 2018, 6:14pm

You will need two filters mutate filter with the split function to change the fields string value into an array of strings and split filter to split the one event into 3 (or more) based on the newly created array.
Example:

input {
  generator { count => 1 message => "RADIUS: MSCHAP: AD status:Logon failure (0xc000006d)
MSCHAP: Authentication failed
EAP-MSCHAPv2: User authentication failure" }
}

filter {
  # we can do this in one mutate filter block
  # because the gsub is done before the split in the code.
  # we have to use gsub because the mutate split does not use regular expressions
  # we use "~^^ø" because it is an improbable sequence of
  # characters, unlikely to be found in a value and looks funny!
  mutate {
    gsub => ["[message]", "\s?\n", "~^^ø"]
    split => { "[message]" => "~^^ø" } # creates an array of strings in the [message] field]
  }
  split { field => "[message]" }
}

output { stdout { codec => rubydebug } }

Result:

{
    "@timestamp" => 2018-09-26T18:00:59.579Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "sequence" => 0,
       "message" => "RADIUS: MSCHAP: AD status:Logon failure (0xc000006d)"
}
{
    "@timestamp" => 2018-09-26T18:00:59.579Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "sequence" => 0,
       "message" => "MSCHAP: Authentication failed"
}
{
    "@timestamp" => 2018-09-26T18:00:59.579Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "sequence" => 0,
       "message" => "EAP-MSCHAPv2: User authentication failure"
}

Obviously, once you move this to your config you need to use the [Common.Alerts] field instead of [message]

Dr_Natas · September 26, 2018, 6:42pm

Thank you, I'll let you know of my progress.

Dr_Natas · September 27, 2018, 5:59pm

So far this is what I was able to do with your help.

{
"615 10.30.42.8 server-name 270369 1 0 Common.Username" => "john.doe",
                          "Common.System-Posture-Token" => "UNKNOWN",
                                    "Common.Error-Code" => "216",
                                        "unknown_data3" => "0",
                                                 "time" => [
    [0] "14:47:22",
    [1] "47:22,559"
],
                                       "Common.Service" => "WLAN_Enterprise",
                                         "Common.Roles" => "Guest",
                                           "@timestamp" => 2018-09-27T18:01:16.440Z,
                                           "log_number" => "1453",
                                   "RADIUS.Auth-Method" => "EAP-PEAP",
                              "Common.Host-MAC-Address" => "deadbeefdead",
                                        "syslog_server" => "syslog-server",
                                   "RADIUS.Auth-Source" => "AD:three.fifty.com",
                                                 "host" => "pcmastrace",
                             "Common.Request-Timestamp" => "2018-09-25 14:50:17-07",
                                        "Common.Alerts" => [
    [0] "RADIUS: MSCHAP: AD status:Logon failure (0xc000006d) \\nMSCHAP: Authentication failed\\nEAP-MSCHAPv2: User authentication failure"
],
                                                 "date" => "2018-09-25",
                                         "Clearpass_IP" => "10.30.42.9",
                                "Common.NAS-IP-Address" => "1.2.3.4",
                                        "unknown_data2" => "1",
                                         "unknown_data" => "14:",
                                             "MonthDay" => "25",
                          "Common.Enforcement-Profiles" => "PAN-Update-Node",
                                             "sequence" => 0,
                                             "@version" => "1",
                                                "Month" => "Sep"
}

This is what I was looking into.
I saw how your brilliant code was able to create an Array for my [Common.Alerts] filed. I was talking to my co-worker about this and he said that we can query data from what we got so far. He was "good enough", but I'm still curious if I can a split the field "Common.Alerts" up to the \n. Then add a new field e.g. "Common.Alerts2" up to the next \n, up until the last \n. Sorry for all the trouble this my hello world introduction to logstash.

Split filter plugin | Logstash Reference [6.4] | Elastic

input {
  generator { count => 1 message => "Sep 25 14:47:22 2018-09-25 14: 47:22,559 10.30.42.9 syslog-server 1453 1 0 Sep 25 14:51:20 2018-09-25 14: 51:20,615 10.30.42.8 server-name 270369 1 0 Common.Username=john.doe,Common.Service=WLAN_Enterprise,Common.Roles=[Guest],RADIUS.Auth-Source=AD:three.fifty.com,RADIUS.Auth-Method=EAP-PEAP,EAP-MSCHAPv2,Common.System-Posture-Token=UNKNOWN,Common.Enforcement-Profiles=PAN-Update-Node, cci_connect Access Profile,Common.Host-MAC-Address=deadbeefdead,Common.NAS-IP-Address=1.2.3.4,Common.Error-Code=216,Common.Alerts=RADIUS: MSCHAP: AD status:Logon failure (0xc000006d) \nMSCHAP: Authentication failed\nEAP-MSCHAPv2: User authentication failure,Common.Request-Timestamp=2018-09-25 14:50:17-07" }
}

filter {
grok {
    match => {
        "message" => "%{MONTH:Month}%{SPACE}%{MONTHDAY:MonthDay}%{SPACE}%{TIME:time}%{SPACE}(?<date>\d{4}-\d{2}-\d{2})%{SPACE} %{DATA:unknown_data} %{DATA:time} %{IPV4:Clearpass_IP} %{SYSLOGHOST:syslog_server} %{NUMBER:log_number} %{INT:unknown_data2} %{INT:unknown_data3} %{GREEDYDATA:kv_pairs}"
    }
}   

kv {
    source => "kv_pairs"
    field_split => "," 
}   

mutate {
    gsub => ["[Common.Alerts]", "\s?\n", "~^^ø"]
    split => { "[Common.Alerts]" => "~^^ø" } # creates an array of strings in the [message] field]
}   

mutate {
    # Original message has been fully parsed, so remove it. 
    remove_field => [ "message" ]
    # Remove kv_pairs to avoid duplicate logs
    remove_field => [ "kv_pairs" ]
}   

}
output { stdout { codec => rubydebug } }

Dr_Natas · September 27, 2018, 10:39pm

I got it working!! Learning the syntax for Logstash is a challenge but it has a great rewards. Got to love Arrays.

    mutate {
      gsub => ["[Common.Alerts]", "\\n", ","]
      split => { "[Common.Alerts]" => "," } # creates an array of strings in the [message] field]
      add_field => {
         "RADIUS" => "%{[Common.Alerts][0]}"
         "MSCHAP"  => "%{[Common.Alerts][1]}"
          }
  }

system · October 25, 2018, 10:39pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.