# Clearpass logs \[SOLVED\] **URL:** \ **Category:** Logstash\ **Created:** [September 25, 2018, 9:41pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906 "2018-09-25T21:41:08Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Dr\_Natas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dr_natas/32/36046_2.png) [@Dr\_Natas](https://discuss.elastic.co/u/Dr_Natas)\ **Post date:** [September 25, 2018, 9:41pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906/1 "2018-09-25T21:41:09Z")
![LOGS](https://us1.discourse-cdn.com/elastic/original/3X/3/c/3c1095be0889492fe6c0ee214046a0025b5a9005.png) This is my first grok filter, if any one has any input I'm more than welcome to receive any. I'm able to parse a good portion of Aruba Clearpass logs but this field I'm unable split it by columns e.g. I would like to split that row into what we see bellow. I'm running Logstash 6.4, and I also included a picture of how the row looks. Bellow is my 1st grok filter. RADIUS: MSCHAP: AD status:Logon failure (0xc000006d) \nMSCHAP: Authentication failed\nEAP-MSCHAPv2: User authentication failure RADIUS : MSCHAP AD status : Login failure (0xc000006d) MSCHAP: Authentication failed ... ``` filter { if "aruba" in [tags] { grok { match => { "message" => "%{MONTH:Month}%{SPACE}%{MONTHDAY:MonthDay}%{SPACE}%{TIME:time}%{SPACE}(?\d{4}-\d{2}-\d{2})%{SPACE} %{DATA:unknown_data} %{DATA:time} %{IPV4:Clearpass_IP} %{SYSLOGHOST:syslog_server} %{NUMBER:log_number} %{INT:unknown_data2} %{INT:unknown_data3} %{GREEDYDATA:kv_pairs}" } } kv { source => "kv_pairs" field_split => "," } mutate { # Original message has been fully parsed, so remove it. remove_field => ["message"] # Remove kv_pairs to avoid duplicate logs remove_field => ["kv_pairs"] } } } ``` [Log Sample] Sep 25 14:47:22 2018-09-25 14: 47:22,559 10.30.42.9 syslog-server 1453 1 0 Sep 25 14:51:20 2018-09-25 14: 51:20,615 10.30.42.8 server-name 270369 1 0 Common.Username=john.doe,Common.Service=WLAN\_Enterprise,Common.Roles=[Guest],RADIUS.Auth-Source=AD:three.fifty.com,RADIUS.Auth-Method=EAP-PEAP,EAP-MSCHAPv2,Common.System-Posture-Token=UNKNOWN,Common.Enforcement-Profiles=PAN-Update-Node, cci\_connect Access Profile,Common.Host-MAC-Address=deadbeefdead,Common.NAS-IP-Address=1.2.3.4,Common.Error-Code=216,Common.Alerts=RADIUS: MSCHAP: AD status:Logon failure (0xc000006d) \nMSCHAP: Authentication failed\nEAP-MSCHAPv2: User authentication failure,Common.Request-Timestamp=2018-09-25 14:50:17-07 I took some of my ideas from his guy. > ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [September 26, 2018, 12:27pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906/2 "2018-09-26T12:27:36Z")
> [@Dr\_Natas](#): > > RADIUS : MSCHAP > AD status : Login failure (0xc000006d) > MSCHAP: Authentication failed You need to be more specific about the above. Do you want three separate events with the `Common.Alerts` field set to each line above, but all the rest of the fields duplicated? ---
**Author:** ![Dr\_Natas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dr_natas/32/36046_2.png) [@Dr\_Natas](https://discuss.elastic.co/u/Dr_Natas)\ **Post date:** [September 26, 2018, 3:51pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906/3 "2018-09-26T15:51:38Z")
Yes, I believe having three separate events would be perfect and would make queering data much better. I think if I'm able to create custom fields would be better. I'm new to this but I hope my mistakes would enlighten the next person. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [September 26, 2018, 6:14pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906/4 "2018-09-26T18:14:37Z")
You will need two filters `mutate filter with the split function` to change the fields string value into an array of strings and `split filter` to split the one event into 3 (or more) based on the newly created array. Example: ```auto input { generator { count => 1 message => "RADIUS: MSCHAP: AD status:Logon failure (0xc000006d) MSCHAP: Authentication failed EAP-MSCHAPv2: User authentication failure" } } filter { # we can do this in one mutate filter block # because the gsub is done before the split in the code. # we have to use gsub because the mutate split does not use regular expressions # we use "~^^ø" because it is an improbable sequence of # characters, unlikely to be found in a value and looks funny! mutate { gsub => ["[message]", "\s?\n", "~^^ø"] split => { "[message]" => "~^^ø" } # creates an array of strings in the [message] field] } split { field => "[message]" } } output { stdout { codec => rubydebug } } ``` Result: ```auto { "@timestamp" => 2018-09-26T18:00:59.579Z, "@version" => "1", "host" => "Elastics-MacBook-Pro.local", "sequence" => 0, "message" => "RADIUS: MSCHAP: AD status:Logon failure (0xc000006d)" } { "@timestamp" => 2018-09-26T18:00:59.579Z, "@version" => "1", "host" => "Elastics-MacBook-Pro.local", "sequence" => 0, "message" => "MSCHAP: Authentication failed" } { "@timestamp" => 2018-09-26T18:00:59.579Z, "@version" => "1", "host" => "Elastics-MacBook-Pro.local", "sequence" => 0, "message" => "EAP-MSCHAPv2: User authentication failure" } ``` Obviously, once you move this to your config you need to use the `[Common.Alerts]` field instead of `[message]` ---
**Author:** ![Dr\_Natas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dr_natas/32/36046_2.png) [@Dr\_Natas](https://discuss.elastic.co/u/Dr_Natas)\ **Post date:** [September 26, 2018, 6:42pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906/5 "2018-09-26T18:42:41Z")
Thank you, I'll let you know of my progress. ---
**Author:** ![Dr\_Natas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dr_natas/32/36046_2.png) [@Dr\_Natas](https://discuss.elastic.co/u/Dr_Natas)\ **Post date:** [September 27, 2018, 5:59pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906/6 "2018-09-27T17:59:19Z")
> [@guyboertje](#): > > ```auto > > ``` So far this is what I was able to do with your help. ``` { "615 10.30.42.8 server-name 270369 1 0 Common.Username" => "john.doe", "Common.System-Posture-Token" => "UNKNOWN", "Common.Error-Code" => "216", "unknown_data3" => "0", "time" => [ [0] "14:47:22", [1] "47:22,559" ], "Common.Service" => "WLAN_Enterprise", "Common.Roles" => "Guest", "@timestamp" => 2018-09-27T18:01:16.440Z, "log_number" => "1453", "RADIUS.Auth-Method" => "EAP-PEAP", "Common.Host-MAC-Address" => "deadbeefdead", "syslog_server" => "syslog-server", "RADIUS.Auth-Source" => "AD:three.fifty.com", "host" => "pcmastrace", "Common.Request-Timestamp" => "2018-09-25 14:50:17-07", "Common.Alerts" => [ [0] "RADIUS: MSCHAP: AD status:Logon failure (0xc000006d) \\nMSCHAP: Authentication failed\\nEAP-MSCHAPv2: User authentication failure" ], "date" => "2018-09-25", "Clearpass_IP" => "10.30.42.9", "Common.NAS-IP-Address" => "1.2.3.4", "unknown_data2" => "1", "unknown_data" => "14:", "MonthDay" => "25", "Common.Enforcement-Profiles" => "PAN-Update-Node", "sequence" => 0, "@version" => "1", "Month" => "Sep" } ``` * * * This is what I was looking into. I saw how your brilliant code was able to create an Array for my [Common.Alerts] filed. I was talking to my co-worker about this and he said that we can query data from what we got so far. He was "good enough", but I'm still curious if I can a split the field "Common.Alerts" up to the \n. Then add a new field e.g. "Common.Alerts2" up to the next \n, up until the last \n. Sorry for all the trouble this my hello world introduction to logstash. ## [Split filter plugin | Logstash Reference [6.4] | Elastic](https://www.elastic.co/guide/en/logstash/6.4/plugins-filters-split.html) ``` input { generator { count => 1 message => "Sep 25 14:47:22 2018-09-25 14: 47:22,559 10.30.42.9 syslog-server 1453 1 0 Sep 25 14:51:20 2018-09-25 14: 51:20,615 10.30.42.8 server-name 270369 1 0 Common.Username=john.doe,Common.Service=WLAN_Enterprise,Common.Roles=[Guest],RADIUS.Auth-Source=AD:three.fifty.com,RADIUS.Auth-Method=EAP-PEAP,EAP-MSCHAPv2,Common.System-Posture-Token=UNKNOWN,Common.Enforcement-Profiles=PAN-Update-Node, cci_connect Access Profile,Common.Host-MAC-Address=deadbeefdead,Common.NAS-IP-Address=1.2.3.4,Common.Error-Code=216,Common.Alerts=RADIUS: MSCHAP: AD status:Logon failure (0xc000006d) \nMSCHAP: Authentication failed\nEAP-MSCHAPv2: User authentication failure,Common.Request-Timestamp=2018-09-25 14:50:17-07" } } filter { grok { match => { "message" => "%{MONTH:Month}%{SPACE}%{MONTHDAY:MonthDay}%{SPACE}%{TIME:time}%{SPACE}(?\d{4}-\d{2}-\d{2})%{SPACE} %{DATA:unknown_data} %{DATA:time} %{IPV4:Clearpass_IP} %{SYSLOGHOST:syslog_server} %{NUMBER:log_number} %{INT:unknown_data2} %{INT:unknown_data3} %{GREEDYDATA:kv_pairs}" } } kv { source => "kv_pairs" field_split => "," } mutate { gsub => ["[Common.Alerts]", "\s?\n", "~^^ø"] split => { "[Common.Alerts]" => "~^^ø" } # creates an array of strings in the [message] field] } mutate { # Original message has been fully parsed, so remove it. remove_field => ["message"] # Remove kv_pairs to avoid duplicate logs remove_field => ["kv_pairs"] } } output { stdout { codec => rubydebug } } ``` ---
**Author:** ![Dr\_Natas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dr_natas/32/36046_2.png) [@Dr\_Natas](https://discuss.elastic.co/u/Dr_Natas)\ **Post date:** [September 27, 2018, 10:39pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906/7 "2018-09-27T22:39:10Z")
I got it working!! Learning the syntax for Logstash is a challenge but it has a great rewards. Got to love Arrays. 🙂 ``` mutate { gsub => ["[Common.Alerts]", "\\n", ","] split => { "[Common.Alerts]" => "," } # creates an array of strings in the [message] field] add_field => { "RADIUS" => "%{[Common.Alerts][0]}" "MSCHAP" => "%{[Common.Alerts][1]}" } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 25, 2018, 10:39pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906/8 "2018-10-25T22:39:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.