# Parse Exchange Message tracking logs via Dissect filebeat processor **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [January 4, 2022, 12:35pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445 "2022-01-04T12:35:55Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![a.gavric](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@a.gavric](https://discuss.elastic.co/u/a.gavric)\ **Post date:** [January 4, 2022, 12:35pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/1 "2022-01-04T12:35:55Z")
Hello everyone i have been trying to ingest and parse Exchange message tracking logs via filebeat dissect processor to generate fields i need from the log rather than the beat dumping the entire log into the message field and so far i had no luck with it.as an example this is how the log file looks like: ```auto #Software: Microsoft Exchange Server #Version: 15.01.2176.014 #Log-type: Message Tracking Log #Date: 2021-11-28T15:44:24.950Z #Fields: date-time,client-ip,client-hostname,server-ip,server-hostname,source-context,connector-id,source,event-id,internal-message-id,message-id,network-message-id,recipient-address,recipient-status,total-bytes,recipient-count,related-recipient-address,reference,message-subject,sender-address,return-path,message-info,directionality,tenant-id,original-client-ip,original-server-ip,custom-data,transport-traffic-type,log-id,schema-version 2021-11-28T15:44:24.950Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,86b9db72-80a9-4d15-f9b6-08d9b285f458,15.01.2176.014 2021-11-28T15:44:24.959Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,d149c2e5-09d6-474d-5170-08d9b285f45a,15.01.2176.014 2021-11-28T15:44:29.828Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,32b7c2a6-6dfe-4a5f-49ff-08d9b285f741,15.01.2176.014 2021-11-28T15:44:35.913Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,19fc7985-b9bf-41fb-c927-08d9b285fae1,15.01.2176.014 ``` I have been trying to create fields in elasticsearch of of the #Fields column in the log only and have it split the message into those fields this it the processor i have been trying to use: ```auto processors: - dissect: tokenizer: "%{date-time} %{client-ip} %{client-hostname} %{server-ip} %{server-hostname} %{source-context} %{connector-id} %{source} %{event-id} %{internal-message-id} , %{message-id} , %{network-message-id} , %{recipient-address} %{recipient-status} %{total-bytes|integer} %{recipient-count|integer} %{related-recipient-address} %{reference} %{message-subject} %{sender-address} %{return-path} %{message-info} %{directionality} %{tenant-id} %{original-client-ip} %{original-server-ip} %{custom-data} %{transport-traffic-type} %{log-id} %{schema-version}" field: "message" target_prefix: "" ignore_failure: "true" overwrite_keys: "true" ``` but when ingesting im only getting > log.flags: dissect\_parsing\_error Any help would be appreciated as i am relatively new with using processors and data enrichment. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 4, 2022, 1:04pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/2 "2022-01-04T13:04:03Z")
The fields in your messages are separated by commas, but your tokenizer is separating them with spaces, this won't match. You need to change your tokenizer to use comma instead of space. Something like this: ```auto tokenizer: "%{date-time},%{client-ip},%{client-hostname}, ... ``` ---
**Author:** ![a.gavric](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@a.gavric](https://discuss.elastic.co/u/a.gavric)\ **Post date:** [January 4, 2022, 1:05pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/3 "2022-01-04T13:05:34Z")
> [@leandrojmp](#): > > `%{date-time},` Ive tried that first and was still getting the same error, let me look through my previous tokenizer and see if i made a mistake somewhere ---
**Author:** ![a.gavric](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@a.gavric](https://discuss.elastic.co/u/a.gavric)\ **Post date:** [January 4, 2022, 1:20pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/4 "2022-01-04T13:20:10Z")
Hi Leandro even when using commas as a separator it is still comming back with the same error message. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 4, 2022, 1:29pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/5 "2022-01-04T13:29:14Z")
Can you share the full log you are receiving? I do not use filebeat much to parse messages, but maybe there is some hint of the issue in your log. You are using the following tokenizer, right? ```auto %{date-time},%{client-ip},%{client-hostname},%{server-ip},%{server-hostname},%{source-context},%{connector-id},%{source},%{event-id},%{internal-message-id},%{message-id},%{network-message-id},%{recipient-address},%{recipient-status},%{total-bytes|integer},%{recipient-count|integer},%{related-recipient-address},%{reference},%{message-subject},%{sender-address},%{return-path},%{message-info},%{directionality},%{tenant-id},%{original-client-ip},%{original-server-ip},%{custom-data},%{transport-traffic-type},%{log-id},%{schema-version} ``` ---
**Author:** ![a.gavric](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@a.gavric](https://discuss.elastic.co/u/a.gavric)\ **Post date:** [January 4, 2022, 1:40pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/6 "2022-01-04T13:40:07Z")
Hi Leandro this is the full log i am receiving: ```auto { "_index": "filebeat-7.16.2-2022.01.04-000001", "_type": "_doc", "_id": "uVVLJX4BOAStKFAz9Npo", "_version": 1, "_score": 1, "_source": { "@timestamp": "2022-01-04T13:33:59.294Z", "log": { "flags": [ "dissect_parsing_error" ], "offset": 1418, "file": { "path": "C:\\Program Files\\Microsoft\\Exchange Server\\V15\\TransportRoles\\Logs\\MessageTracking\\MSGTRK2021122323-1.LOG" } }, "message": "2021-12-23T23:02:27.225Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,2ee5ccf5-2ed3-4c72-9d8e-08d9c6684a21,15.01.2176.014", "input": { "type": "filestream" }, "agent": { "ephemeral_id": "55bdf61a-c428-4be1-b9ac-886c57b12592", "id": "1b96a4c0-fb4d-4ce9-88ba-6dd4d87d7b07", "name": "EXCHANGE01", "type": "filebeat", "version": "7.16.2", "hostname": "EXCHANGE01" }, "ecs": { "version": "1.12.0" }, "host": { "hostname": "EXCHANGE01", "architecture": "x86_64", "os": { "kernel": "10.0.14393.4583 (rs1_release.210730-1850)", "build": "14393.4583", "type": "windows", "platform": "windows", "version": "10.0", "family": "windows", "name": "Windows Server 2016 Datacenter" }, "id": "2c88d262-451b-4a93-bcb6-a22c67a5b89e", "ip": [ "fe80::51d0:d0c1:cd4:3a67", "172.172.31.30" ], "name": "EXCHANGE01", "mac": [ "52:54:00:b9:49:4d" ] } }, "fields": { "host.os.name.text": [ "Windows Server 2016 Datacenter" ], "host.hostname": [ "EXCHANGE01" ], "host.mac": [ "52:54:00:b9:49:4d" ], "host.os.build": [ "14393.4583" ], "host.ip": [ "fe80::51d0:d0c1:cd4:3a67", "172.172.31.30" ], "agent.type": [ "filebeat" ], "host.os.version": [ "10.0" ], "host.os.kernel": [ "10.0.14393.4583 (rs1_release.210730-1850)" ], "host.os.name": [ "Windows Server 2016 Datacenter" ], "agent.name": [ "EXCHANGE01" ], "host.name": [ "EXCHANGE01" ], "host.id": [ "2c88d262-451b-4a93-bcb6-a22c67a5b89e" ], "host.os.type": [ "windows" ], "input.type": [ "filestream" ], "log.offset": [ 1418 ], "log.flags": [ "dissect_parsing_error" ], "agent.hostname": [ "EXCHANGE01" ], "message": [ "2021-12-23T23:02:27.225Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,2ee5ccf5-2ed3-4c72-9d8e-08d9c6684a21,15.01.2176.014" ], "host.architecture": [ "x86_64" ], "@timestamp": [ "2022-01-04T13:33:59.294Z" ], "agent.id": [ "1b96a4c0-fb4d-4ce9-88ba-6dd4d87d7b07" ], "ecs.version": [ "1.12.0" ], "host.os.platform": [ "windows" ], "log.file.path": [ "C:\\Program Files\\Microsoft\\Exchange Server\\V15\\TransportRoles\\Logs\\MessageTracking\\MSGTRK2021122323-1.LOG" ], "agent.ephemeral_id": [ "55bdf61a-c428-4be1-b9ac-886c57b12592" ], "agent.version": [ "7.16.2" ], "host.os.family": [ "windows" ] } } ``` It splits 1 log file into 4 separate json messages below is the log file in its .log format: ```auto #Software: Microsoft Exchange Server #Version: 15.01.2176.014 #Log-type: Message Tracking Log #Date: 2021-12-23T23:02:22.040Z #Fields: date-time,client-ip,client-hostname,server-ip,server-hostname,source-context,connector-id,source,event-id,internal-message-id,message-id,network-message-id,recipient-address,recipient-status,total-bytes,recipient-count,related-recipient-address,reference,message-subject,sender-address,return-path,message-info,directionality,tenant-id,original-client-ip,original-server-ip,custom-data,transport-traffic-type,log-id,schema-version 2021-12-23T23:02:22.040Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,0eefcc2e-b433-4bc1-8194-08d9c668470a,15.01.2176.014 2021-12-23T23:02:22.814Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,1fe8f900-adb1-441a-abed-08d9c6684780,15.01.2176.014 2021-12-23T23:02:26.711Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,7bb25f3b-76ab-4363-395a-08d9c66849d3,15.01.2176.014 2021-12-23T23:02:27.225Z,172.172.254.41,EXCHANGE01,172.172.31.30,,Recipient Filter Agent,,SMTP,FAIL,,,,,,,,,,,,,,,,,,S:ToEntity=Unknown;S:FromEntity=Unknown,,2ee5ccf5-2ed3-4c72-9d8e-08d9c6684a21,15.01.2176.014 ``` ---
**Author:** ![a.gavric](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@a.gavric](https://discuss.elastic.co/u/a.gavric)\ **Post date:** [January 4, 2022, 1:45pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/7 "2022-01-04T13:45:50Z")
Hi Leandro, i got it to work it was a missed comma in the tokenizer. Thank you so much! ---
**Author:** ![a.gavric](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@a.gavric](https://discuss.elastic.co/u/a.gavric)\ **Post date:** [January 4, 2022, 1:49pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/8 "2022-01-04T13:49:33Z")
As an addition this is how the configuration in the beat looks like now ```auto - dissect: tokenizer: '%{date-time},%{client-ip},%{client-hostname},%{server-ip},%{server-hostname},%{source-context},%{connector-id},%{source},%{event-id},%{internal-message-id},%{message-id},%{network-message-id},%{recipient-address},%{recipient-status},%{total-bytes},%{recipient-count},%{related-recipient-address},%{reference},%{message-subject},%{sender-address},%{return-path},%{message-info},%{directionality},%{tenant-id},%{original-client-ip},%{original-server-ip},%{custom-data},%{transport-traffic-type},%{log-id},%{schema-version}' field: "message" target_prefix: "dissect" ``` ---
**Author:** ![a.gavric](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@a.gavric](https://discuss.elastic.co/u/a.gavric)\ **Post date:** [January 4, 2022, 2:06pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/9 "2022-01-04T14:06:47Z")
This can be marked as solved, thank you again. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 1, 2022, 4:07pm UTC](https://discuss.elastic.co/t/parse-exchange-message-tracking-logs-via-dissect-filebeat-processor/293445/10 "2022-02-01T16:07:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.