Parsing Alcatel PBX multiline log tickets

Hi,

I am trying to parse call trace logs from an Alcatel PBX and could not find any examples on it being done.

I found how to pick up the complete tickets using multiline.
The messages vary, all messages have a header field and most but not all an additional data field.
The header and data fields are not identical in the different tickets.

Three example tickets:
Message 1:


| (045925:000002) Concatenated-Physical-Event :
| long: 56 desti: 0 source: 0 cryst: 1 cpl: 2 us: 0 term: 0 type a5
| tei: 0 >>>> message received : SETUP [05] Call ref : 31 6f
| SENDING COMPLETE
|______________________________________________________________________________
|
| IE:[04] BEARER_CAPABILITY (l=3) 90 90 a3
| IE:[18] CHANNEL (l=3) a9 83 81 -> T2 : B channel 1 exclusive
| IE:[1e] PROGRESS_ID (l=2) 81 83
| IE:[6c] CALLING_NUMBER (l=9) -> 80 Num : 12345678
| IE:[70] CALLED_NUMBER (l=10) -> 80 Num : 087654321
|______________________________________________________________________________

Message 2:


| (045932:000013) Concatenated-Physical-Event :
| long: 18 desti: 0 source: 0 cryst: 8 cpl: 2 us: 0 term: 0 type a5
| tei: 0 >>>> message received : CONNECT ACK (0f) Call ref : 00 02
|______________________________________________________________________________

Message 3:


| (045948:000014) 1228: Send_IO1 (link-nbr=1, sapi=0, tei=0) :
| long: 55 desti: 0 source: 15 cryst: 7 cpl: 2 us: 8 term: 0 type a5
| tei: 0 <<<< message sent : SETUP [05] Call ref : 00 0b
|______________________________________________________________________________
|
| IE:[04] BEARER_CAPABILITY (l=3) 90 90 a3
| IE:[1e] PROGRESS_ID (l=2) 80 83
| IE:[6c] CALLING_NUMBER (l=10) -> 21 81 Num : 12345678
| IE:[70] CALLED_NUMBER (l=9) -> 81 Num : 87654321
| IE:[7d] HLC (l=2) 91 81
| [a1] Sending complete
|______________________________________________________________________________

I have tried with various stuff but the closest I've ever been is this filter which give me an array containing the ticket in lines.

filter
{

Alcatel PBX Tickets

if [type] == "debug"
{
multiline
{
pattern => "^|"
what => "previous"
}
mutate
{
split => { "message" => "|" }
}
}
}

In bash I would probably have made an array for the header and one for the data and parsed them from there but how do I do this in logstash?

I tried many grok methods but they all crashed because the tickets are different in syntax.
Are there any other filters I should use instead?

Any constructive feedback is welcomed :slight_smile:

Just in case anyone else have the same issue as I had this is the config I ended up with.
Not very elegant I know, but it seem to do the trick.

input
{
tcp
{
port => 11111
type => "alcatel-ticket"
}
}
filter
{
multiline
{
pattern => "^|"
what => "previous"
}
}
filter
{
if [type] == "alcatel-ticket" and "multiline" in [tags] and "+--------+-------+--------+" not in [message]
{
grok
{
match => ["message", "(?m)%{NUMBER:alch_event_uptime}:%{NUMBER:alch_event_no}%{NEXT_PIPE}%{NEXT_COLON}:%{NEXT_COLON}:%{NEXT_COLON}:%{NEXT_COLON}: *%{BASE16NUM:alch_cryst}%{NEXT_COLON}: *%{BASE16NUM:alch_cpl}%{NEXT_ARROW_NOT}%{NOTSPACE:alch_event_direction}%{NEXT_COLON}: *%{NEXT_ALT8_NOT:alch_event_name}%{NEXT_COLON}: *%{BASE16NUM} %{BASE16NUM:alch_callref2}%{NEXT_UNDER}_%{GREEDYDATA:alc_body}"]
remove_tag => ["multiline"]
}
if [alc_body]
{
if "CALLING_NUMBER" in [alc_body]
{
grok
{
match => ["alc_body", "(?m)%{NEXT_CALLING}%{NEXT_COLON}: *%{NUMBER:alcb_number_calling}"]
}
if "_grokparsefailure" in [tags]
{
mutate
{
add_field => { "alcb_number_calling" => "" }
remove_tag => ["_grokparsefailure"]
}
}
}
if "CALLED_NUMBER" in [alc_body]
{
grok
{
match => ["alc_body", "(?m)%{NEXT_CALLED}%{NEXT_COLON}: *%{NUMBER:alcb_number_called}"]
}
if "_grokparsefailure" in [tags]
{
mutate
{
add_field => { "alcb_number_called" => "" }
remove_tag => ["_grokparsefailure"]
}
}
}
if "CONNECTED_NUMBER" in [alc_body]
{
grok
{
match => ["alc_body", "(?m)%{NEXT_CONNECTED}%{NEXT_COLON}: *%{NUMBER:alcb_number_connected}"]
}
if "_grokparsefailure" in [tags]
{
mutate
{
add_field => { "alcb_number_connected" => "" }
remove_tag => ["_grokparsefailure"]
}
}
}
else if "DATE" in [alc_body]
{
grok
{
match => ["alc_body", "(?m)%{NEXT_ARROW_NOT}>%{NEXT_COLON}: *%{NUMBER:alcb_event_date_dd} */ *%{NUMBER:alcb_event_date_mm} */ *%{NUMBER:alcb_event_date_yy} *%{NUMBER:alcb_event_date_HH}:%{NUMBER:alcb_event_date_MM}"]
add_field => { "alcb_event_date" => "20%{alcb_event_date_yy}%{alcb_event_date_mm}%{alcb_event_date_dd}-%{alcb_event_date_HH}%{alcb_event_date_MM}" }
remove_field => ["alc_body", "alcb_event_date_yy", "alcb_event_date_mm", "alcb_event_date_dd", "alcb_event_date_HH", "alcb_event_date_MM"]
}
}
else if "DISCONNECT" in [alch_event_name]
{
grok
{
match => ["alc_body", "(?m)%{NEXT_ARROW_NOT}> *%{NEXT_PIPE:alcb_disconnect_reason}"]
remove_field => ["alc_body"]
}
mutate
{
gsub => ["alcb_disconnect_reason", "[\n]", ""]
}
}
}

# Cleanup
if [alch_event_direction]
{
  if ">>>" in [alch_event_direction]
  {
    mutate
    {
      replace => { "alch_event_direction" => "received" }
    }
  }
  else if "<<<" in [alch_event_direction]
  {
    mutate
    {
      replace => { "alch_event_direction" => "sent" }
    }
  }
}
if [alch_event_name]
{
  mutate
  {
    strip => ["alch_event_name"]
  }
}
mutate
{
  convert => {"alch_event_uptime" => "integer"}
  convert => {"alch_event_no" => "integer"}
}

}
}