Logstash filter: a field pointing to many format of the same field

[Farah_Bhr (Farah Bel Haj Rhouma)]

I have a lot of logs and I want to search through these log lines a callID flow and visualize everything related to that callID into kibana
with logstash I made a filter that detects an hexadecimal format for that CallID
I want to make a filter that points to many formats(decimal, binary..) of a callID so that all these formats point to a same field (a same callID) , so that I can search related to that same callID with kibana
I have 2 issues:

1. my callid is present in many formats , hexadecimal , decimal, binary..
2. I have log lines not containing any callid format , but they are related to that same callid and I want to visualize them , so how can I do to detect that
   Any help please

[leandrojmp (Leandro Pereira)]

You need to provide more context about what you want to do because it is not clear.

What do you mean by many formats? Can you share some log samples and show the expected result?

[Farah_Bhr (Farah Bel Haj Rhouma)]

In the log lines, I can find my callID in many formats, for example, in hexa like this 0xF560032E0001A0000 or in decimal like this 5600000009800000013
Also, I have other log lines not containing any format of this callID but the events are related to the same call and I want to visualize them
Here are some example of log lines:

2021-12-04 22:55:39.532 ServerName SeviceName: [ProcessID] Created CSCall with CallID = 0xF560032E0001A0000
2021-12-04 22:55:39.532 ServerName SeviceName: [ProcessID] ScriptManager::initialize(): script manager has already been initialized
2021-12-04 22:55:39.532 ServerName SeviceName: [ProcessID] CallMediaStream::CallMediaStream() this:0x00555570
2021-12-04 22:55:39.532 ServerName SeviceName: [ProcessID] CallMediaStream::addParticipant(): creating/adding remote party participant instance 0x00555570 for CallIId: 0xF560032E0001A0000
2021-12-04 22:55:39.532 ServerName SeviceName: [ProcessID] ConnectionServer::findCall(0x0000000000000000) wasn't found.
2021-12-04 22:55:39.532 ServerName SeviceName: [ProcessID] ScriptManager::queryScriptProxy, Script cache will be used: false
2021-12-04 22:59:40.724 ServerName SeviceName: [ProcessID]  5600000009800000013
....

The expected result is to visualize all the log lines related to this callID 0xF560032E0001A0000 for example, including the events not containing any callID format like this line "2021-12-04 22:55:39.532 ServerName SeviceName: [ProcessID] ScriptManager::queryScriptProxy, Script cache will be used: false"

[Rios]

Something like this should be helpful.

filter {

    dissect {
      mapping => {
        "message" => "%{timestamp} %{+timestamp} %{servername} %{sevicename}: [%{processid}] %{msg}"
      }
    }

    grok {
      break_on_match => false
      match => {
        "msg" => [" CallID = %{BASE16NUM:phone}",
                 " CallIId: %{BASE16NUM:phone}",
                 "::findCall\(%{BASE16NUM:phone}\)",
				 "^%{SPACE}%{NUMBER:phone}",
				 "%{GREEDYDATA}"
        ]
      }
    }

    if ([phone] =~ /^0x/ ) { # convert phone number to decimal 
      ruby {
        code => "event.set('phone-dec', event.get('phone').to_i(16))"
      }
    }
      date {
        match => ["timestamp", "ISO8601"]
        # target => "timestamp"
        remove_field => [ "timestamp" ]
      }
   prune { blacklist_names => [  "host", "event", "log" ] } # optionally you can remove unnecessary fields
}

Result:

{
       "message" => "2021-12-04 22:55:39.532 ServerName SeviceName: [ProcessID] Created CSCall with CallID = 0xF560032E0001A0000\r",
    "servername" => "ServerName",
     "phone-dec" => 282898170130560843776,
    "sevicename" => "SeviceName",
     "processid" => "ProcessID",
         "phone" => "0xF560032E0001A0000",
           "msg" => "Created CSCall with CallID = 0xF560032E0001A0000\r",
    "@timestamp" => 2021-12-04T14:55:39.532Z
}
{
       "message" => "2021-12-04 22:55:39.532 ServerName SeviceName: [ProcessID] ScriptManager::initialize(): script manager has already been initialized\r",
    "servername" => "ServerName",
    "sevicename" => "SeviceName",
     "processid" => "ProcessID",
           "msg" => "ScriptManager::initialize(): script manager has already been initialized\r",
    "@timestamp" => 2021-12-04T14:55:39.532Z
}
{
       "message" => "2021-12-04 22:59:40.724 ServerName SeviceName: [ProcessID]  5600000009800000013\r",
    "servername" => "ServerName",
    "sevicename" => "SeviceName",
     "processid" => "ProcessID",
         "phone" => "5600000009800000013",
           "msg" => " 5600000009800000013\r",
    "@timestamp" => 2021-12-04T14:59:40.724Z
}

If there is a binary format, to_i(2) in the ruby plugin. For Kibana, just add filtering by phone