Parse Exchange Message tracking logs via Dissect filebeat processor

Elastic Stack Beats
1

Hello everyone i have been trying to ingest and parse Exchange message tracking logs via filebeat dissect processor to generate fields i need from the log rather than the beat dumping the entire log into the message field and so far i had no luck with it.as an example this is how the log file looks like:

#Software: Microsoft Exchange Server
#Version: 15.01.2176.014
#Log-type: Message Tracking Log
#Date: 2021-11-28T15:44:24.950Z
#Fields: date-time,client-ip,client-hostname,server-ip,server-hostname,source-context,connector-id,source,event-id,internal-message-id,message-id,network-message-id,recipient-address,recipient-status,total-bytes,recipient-count,related-recipient-address,reference,message-subject,sender-address,return-path,message-info,directionality,tenant-id,original-client-ip,original-server-ip,custom-data,transport-traffic-type,log-id,schema-version
2021-11-28T15:44:24.950Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,86b9db72-80a9-4d15-f9b6-08d9b285f458,15.01.2176.014
2021-11-28T15:44:24.959Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,d149c2e5-09d6-474d-5170-08d9b285f45a,15.01.2176.014

2021-11-28T15:44:29.828Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,32b7c2a6-6dfe-4a5f-49ff-08d9b285f741,15.01.2176.014
2021-11-28T15:44:35.913Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,19fc7985-b9bf-41fb-c927-08d9b285fae1,15.01.2176.014

I have been trying to create fields in elasticsearch of of the #Fields column in the log only and have it split the message into those fields
this it the processor i have been trying to use:

processors:
    - dissect:
      tokenizer: "%{date-time} %{client-ip} %{client-hostname} %{server-ip} %{server-hostname} %{source-context} %{connector-id} %{source} %{event-id} %{internal-message-id} , %{message-id} , %{network-message-id} , %{recipient-address} %{recipient-status} %{total-bytes|integer} %{recipient-count|integer} %{related-recipient-address} %{reference} %{message-subject} %{sender-address} %{return-path} %{message-info} %{directionality} %{tenant-id} %{original-client-ip} %{original-server-ip} %{custom-data} %{transport-traffic-type} %{log-id} %{schema-version}"
      field: "message"
      target_prefix: ""
      ignore_failure: "true"
      overwrite_keys: "true"

but when ingesting im only getting

log.flags: dissect_parsing_error

Any help would be appreciated as i am relatively new with using processors and data enrichment.

2

The fields in your messages are separated by commas, but your tokenizer is separating them with spaces, this won't match.

You need to change your tokenizer to use comma instead of space.

Something like this:

tokenizer: "%{date-time},%{client-ip},%{client-hostname}, ...
3

Ive tried that first and was still getting the same error, let me look through my previous tokenizer and see if i made a mistake somewhere

4

Hi Leandro even when using commas as a separator it is still comming back with the same error message.

5

Can you share the full log you are receiving? I do not use filebeat much to parse messages, but maybe there is some hint of the issue in your log.

You are using the following tokenizer, right?

%{date-time},%{client-ip},%{client-hostname},%{server-ip},%{server-hostname},%{source-context},%{connector-id},%{source},%{event-id},%{internal-message-id},%{message-id},%{network-message-id},%{recipient-address},%{recipient-status},%{total-bytes|integer},%{recipient-count|integer},%{related-recipient-address},%{reference},%{message-subject},%{sender-address},%{return-path},%{message-info},%{directionality},%{tenant-id},%{original-client-ip},%{original-server-ip},%{custom-data},%{transport-traffic-type},%{log-id},%{schema-version}
6

Hi Leandro this is the full log i am receiving:

{

  "_index": "filebeat-7.16.2-2022.01.04-000001",

  "_type": "_doc",

  "_id": "uVVLJX4BOAStKFAz9Npo",

  "_version": 1,

  "_score": 1,

  "_source": {

    "@timestamp": "2022-01-04T13:33:59.294Z",

    "log": {

      "flags": [

        "dissect_parsing_error"

      ],

      "offset": 1418,

      "file": {

        "path": "C:\\Program Files\\Microsoft\\Exchange Server\\V15\\TransportRoles\\Logs\\MessageTracking\\MSGTRK2021122323-1.LOG"

      }

    },

    "message": "2021-12-23T23:02:27.225Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,2ee5ccf5-2ed3-4c72-9d8e-08d9c6684a21,15.01.2176.014",

    "input": {

      "type": "filestream"

    },

    "agent": {

      "ephemeral_id": "55bdf61a-c428-4be1-b9ac-886c57b12592",

      "id": "1b96a4c0-fb4d-4ce9-88ba-6dd4d87d7b07",

      "name": "EXCHANGE01",

      "type": "filebeat",

      "version": "7.16.2",

      "hostname": "EXCHANGE01"

    },

    "ecs": {

      "version": "1.12.0"

    },

    "host": {

      "hostname": "EXCHANGE01",

      "architecture": "x86_64",

      "os": {

        "kernel": "10.0.14393.4583 (rs1_release.210730-1850)",

        "build": "14393.4583",

        "type": "windows",

        "platform": "windows",

        "version": "10.0",

        "family": "windows",

        "name": "Windows Server 2016 Datacenter"

      },

      "id": "2c88d262-451b-4a93-bcb6-a22c67a5b89e",

      "ip": [

        "fe80::51d0:d0c1:cd4:3a67",

        "172.172.31.30"

      ],

      "name": "EXCHANGE01",

      "mac": [

        "52:54:00:b9:49:4d"

      ]

    }

  },

  "fields": {

    "host.os.name.text": [

      "Windows Server 2016 Datacenter"

    ],

    "host.hostname": [

      "EXCHANGE01"

    ],

    "host.mac": [

      "52:54:00:b9:49:4d"

    ],

    "host.os.build": [

      "14393.4583"

    ],

    "host.ip": [

      "fe80::51d0:d0c1:cd4:3a67",

      "172.172.31.30"

    ],

    "agent.type": [

      "filebeat"

    ],

    "host.os.version": [

      "10.0"

    ],

    "host.os.kernel": [

      "10.0.14393.4583 (rs1_release.210730-1850)"

    ],

    "host.os.name": [

      "Windows Server 2016 Datacenter"

    ],

    "agent.name": [

      "EXCHANGE01"

    ],

    "host.name": [

      "EXCHANGE01"

    ],

    "host.id": [

      "2c88d262-451b-4a93-bcb6-a22c67a5b89e"

    ],

    "host.os.type": [

      "windows"

    ],

    "input.type": [

      "filestream"

    ],

    "log.offset": [

      1418

    ],

    "log.flags": [

      "dissect_parsing_error"

    ],

    "agent.hostname": [

      "EXCHANGE01"

    ],

    "message": [

      "2021-12-23T23:02:27.225Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,2ee5ccf5-2ed3-4c72-9d8e-08d9c6684a21,15.01.2176.014"

    ],

    "host.architecture": [

      "x86_64"

    ],

    "@timestamp": [

      "2022-01-04T13:33:59.294Z"

    ],

    "agent.id": [

      "1b96a4c0-fb4d-4ce9-88ba-6dd4d87d7b07"

    ],

    "ecs.version": [

      "1.12.0"

    ],

    "host.os.platform": [

      "windows"

    ],

    "log.file.path": [

      "C:\\Program Files\\Microsoft\\Exchange Server\\V15\\TransportRoles\\Logs\\MessageTracking\\MSGTRK2021122323-1.LOG"

    ],

    "agent.ephemeral_id": [

      "55bdf61a-c428-4be1-b9ac-886c57b12592"

    ],

    "agent.version": [

      "7.16.2"

    ],

    "host.os.family": [

      "windows"

    ]

  }

}

It splits 1 log file into 4 separate json messages below is the log file in its .log format:

#Software: Microsoft Exchange Server
#Version: 15.01.2176.014
#Log-type: Message Tracking Log
#Date: 2021-12-23T23:02:22.040Z
#Fields: date-time,client-ip,client-hostname,server-ip,server-hostname,source-context,connector-id,source,event-id,internal-message-id,message-id,network-message-id,recipient-address,recipient-status,total-bytes,recipient-count,related-recipient-address,reference,message-subject,sender-address,return-path,message-info,directionality,tenant-id,original-client-ip,original-server-ip,custom-data,transport-traffic-type,log-id,schema-version
2021-12-23T23:02:22.040Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,0eefcc2e-b433-4bc1-8194-08d9c668470a,15.01.2176.014
2021-12-23T23:02:22.814Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,1fe8f900-adb1-441a-abed-08d9c6684780,15.01.2176.014
2021-12-23T23:02:26.711Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,7bb25f3b-76ab-4363-395a-08d9c66849d3,15.01.2176.014
2021-12-23T23:02:27.225Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,2ee5ccf5-2ed3-4c72-9d8e-08d9c6684a21,15.01.2176.014
7

Hi Leandro, i got it to work it was a missed comma in the tokenizer. Thank you so much!

1 Like
8

As an addition this is how the configuration in the beat looks like now

- dissect:

      tokenizer: '%{date-time},%{client-ip},%{client-hostname},%{server-ip},%{server-hostname},%{source-context},%{connector-id},%{source},%{event-id},%{internal-message-id},%{message-id},%{network-message-id},%{recipient-address},%{recipient-status},%{total-bytes},%{recipient-count},%{related-recipient-address},%{reference},%{message-subject},%{sender-address},%{return-path},%{message-info},%{directionality},%{tenant-id},%{original-client-ip},%{original-server-ip},%{custom-data},%{transport-traffic-type},%{log-id},%{schema-version}'

      field: "message"

      target_prefix: "dissect"
1 Like
9

This can be marked as solved, thank you again.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.