Parse logs Trent Micro Email Security

Elastic Stack Elasticsearch
1

Hello everyone,

I am currently in the process of normalizing and parsing logs from the manufacturer trend micro email security.

Maybe for you it is obvious but in my case as I am not an expert I must mention that I am working on the .conf file for this parse to be successful but I have had problems to complete this process.

As you can see not all the fields have the field - value structure although this part has not been a problem, the problem I have is that the msg and reason fields have blank spaces and if I use the kv filter in the result it only takes the value up to the first space it finds and obviously I need the complete value.

I have tried several methods but it has not been possible to parse these fields. I appreciate your help or suggestion

for example the msg field only takes "hello" and I need all the complete value ""

Log sample

CEF:0|Trend Micro|TMES|1.0.0.0|500101|CTP_DETECTION|2|rt=2021-02-18 04:05:32 cs2Label=timeOfClick cs2=2021-02-03 23:00:00 
request=http://example.com act=blocked msg=hello this is a mail test cs1Label=messageId cs1=<202102181642138223747@trend.com> 
suser=user1@example1.com duser=user2@example2.com```
2

How are you receving those logs? You didn't mention which tools and their versions.

It is Logstash? Filebeat? Elastic Agent?

1 Like
3

I understand that they are being displayed on screen to ensure that they are properly parsed before ingesting them. I don't know if this answers the question (Syslog I would think).

Thank you in advance for your response.

input { tcp { port => 11111 } }

filter {
  dissect {
    mapping => {
      "message" => "<%{pri}> %{timestamp} %{hostname} %{app}[%{pid}]: %{cef_data}"
    }
  }

  mutate {
    replace => { "message" => "%{cef_data}" }
  }

  grok {
    match => {
      "cef_data" => "CEF:%{INT:cef_version}\|%{DATA:device_vendor}\|%{DATA:device_product}\|%{DATA:device_version}\|%{DATA:signature_id}\|%{DATA:name}\|%{INT
:severity}\|%{GREEDYDATA:extensiones}"
    }
  }

  mutate {
    gsub => [
      "rt", " ", "_",
      "suser", " ", "_",
      "duser", " ", "_",
      "msg", " ", "_",
      "src", " ", "_",
      "deviceTranslatedAddress", " ", "_",
      "cs1Label", " ", "_",
      "cs1", " ", "_",
      "cs2Label", " ", "_",
      "cs2", " ", "_",
      "cs3Label", " ", "_",
      "cs3", " ", "_",
      "cn1Label", " ", "_",
      "cn1", " ", "_",
      "reason", " ", "_",
      "act", " ", "_",
      "cs4Label", " ", "_",
      "cs4", " ", "_",
      "cs5Label", " ", "_",
      "cs5", " ", "_"
    ]
  }

  kv {
    source => "extensiones"
    trim_value => " "
    field_split => " "
    value_split => "="
    include_keys => [
      "rt", "suser", "duser", "msg", "src", "deviceTranslatedAddress",
      "cs1Label", "cs1", "cs2Label", "cs2", "cs3Label", "cs3", "cn1Label",
      "cn1", "reason", "act", "cs4Label", "cs4", "cs5Label", "cs5"
    ]
  }

  mutate {
    remove_field => [ "message", "extensiones", "cef_data" ]
  }
}

output { stdout { codec => rubydebug } }
4

Your log is a CEF message, if can use the cef codec to help parse it.

Try this:

input { 
  tcp { 
    port => 11111 
    codec => cef
  } 
}

output { 
  stdout { 
    codec => rubydebug 
  } 
}

This will parse your initial message, see the result and then create extra filters if needed.

2 Likes
5

thank you I will give it a try. Last question the CEF codec performance may vary from a paid version to a free one? I mean if it works better one than the other?

6

No, there is no difference, the paid tiers of the license add extra features, but there will be no difference on performance for any tool.

1 Like