Newb Grok parsing failures, teach me how to fish?

Elastic Stack Logstash
1

Ok, I am fighting with a logfile that I really need to parse from my Radiator server. My problem is that it does not output in a "standard" format and can change. I just want to scan and pull values into fields as it finds them if possible. I tried using the online grok filter builder but I have no clue how it works so if someone could give an example I would greatly appreciate it. In regex with the /g option my filter works just fine however this seems to grab the first line of Code: and everything beyond.

Here is my filter that I am trying to use so you can see what I am trying to snag.

filter {
if [type] == "radius" {
grok {
match => {"message" => "(?m)^Code:\s*(?<RadCode>.*)|^Identifier:\s(?<RadIdent>\d{1,4})|^\s*NAS-Identifier =   \"(?<RadBranch>.*)\"|^\s*User-Name = \"(?<RadUser>.*)\"|^.*Reply-Message = \"(?<RadReply>.*)\""}
named_captures_only => true
    }
  }
}

Here is one of my logs, I combine all lines that have *** from filebeats as a multiline and ship to logstash.

Tue Feb 23 13:58:37 2016: DEBUG: Packet dump:
*** Received from 192.168.xx.xx port 1029 ....
Code:       Access-Request
Identifier: 29
Authentic:  &<123><123><123>}<123><123>_<123><123><123><123>h<123>o<123>
Attributes:
    NAS-Identifier = "RouterName"
    User-Name = "123456789123456789"
    User-Password = <123]<123><123><123><123><123><123><123><123><123>%<123><123><123><123>
    Calling-Station-Id = "123.123.123.123"
    Acct-Session-Id = "00000013"
    Connect-Info = "web-auth"

Tue Feb 23 13:58:37 2016: DEBUG: Handling request with Handler '', Identifier ''
Tue Feb 23 13:58:37 2016: DEBUG:  Deleting session for 12345678912345, 192.168.xx.xx,
Tue Feb 23 13:58:37 2016: DEBUG: Handling with Radius::AuthSIP2:
Tue Feb 23 13:58:37 2016: DEBUG: Radius::AuthSIP2 looks for match with 12345678912345 [12345678912345]
Tue Feb 23 13:58:37 2016: DEBUG: SIP2 send '2300020160223    135837AOXXXX|AA12345678912345|ACterminal password|123456|'
Tue Feb 23 13:58:37 2016: DEBUG: SIP2 read '24YYYY          00020160223       135837AE|AA12345678912345|BLN|AOXXXX|'
Tue Feb 23 13:58:37 2016: DEBUG: Radius::AuthSIP2 REJECT: Bad password: 12345678912345 [12345678912345]
Tue Feb 23 13:58:37 2016: DEBUG: AuthBy SIP2 result: REJECT, Bad password
Tue Feb 23 13:58:37 2016: INFO: Access rejected for 12345678912345: Bad password
Tue Feb 23 13:58:37 2016: DEBUG: do query to 'dbi:mysql:radius:localhost': 'insert into RADAUTHLOG (TIME_STAMP, USERNAME, TYPE, REASON, CLIENT) values (1456264717, '12345678912345', 0, 'Bad password',  '192.168.xx.xx')':
Tue Feb 23 13:58:37 2016: DEBUG: Packet dump:

Here is another example of a combined log

*** Sending to 192.168.xx.xx port 1029 ....
Code:       Access-Reject
Identifier: 29
Authentic:  ej<123><123>f|<123>U ,$*<123><123>5<123>
Attributes:
Reply-Message = "Request Denied"

Is there a better way to do these log files? am I just dumb or missing something? please help.

2

Nevermind I guess, fixed it myself after some digging and trial with lots of errors.

filter {
grok {
break_on_match => false
 match => { message => [ "Code:\s*(?<RadRequest>\w*-\w*).*Code:\s*(?<RadReply>\w*-\w*)" ] }
 match => { message => [ "(?m)^Identifier:\s*(?<RadIdent>\d{1,3})" ] }
 match => { message => [ "(?m)\s*?NAS-Identifier = \"(?<Branch>.*?)\"" ] }
 match => { message => [ "(?m)\s*?User-Name.*\"(?<Patron>.*?)\"" ] }
}
}
1 Like