# Having problem Parsing Message via ingest pipeline **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-alerting\ **Created:** [August 9, 2021, 3:45pm UTC](https://discuss.elastic.co/t/having-problem-parsing-message-via-ingest-pipeline/280827 "2021-08-09T15:45:19Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Alaxwora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alaxwora/32/76976_2.png) [@Alaxwora](https://discuss.elastic.co/u/Alaxwora)\ **Post date:** [August 9, 2021, 3:45pm UTC](https://discuss.elastic.co/t/having-problem-parsing-message-via-ingest-pipeline/280827/1 "2021-08-09T15:45:19Z")
Hello, I was trying to parse CSV files of Message Tracking Logs with filebeat and elasticsearch ingest pipeline, so I used Grok Debugger to create a pattern that matches the message and I had no error then I added the log path of Message tracking logs in filebeat inputs section and added pipeline Id in output.elasticsearch section here is the the error message after running filebeat ![ba69373b307c1ef6466dd78588858397ad096757](https://us1.discourse-cdn.com/elastic/original/3X/5/9/59c11c88f3f4e91cfbd6927c0d1e16ad1f9fcad7.png) Below is the pattern I used in creating pipeline ```auto PUT _ingest/pipeline/GrokTest { "description" : "Convert Exchange csv data to indexed data", "processors" : [ { "grok": { "field": "message", "patterns": ["%{TIMESTAMP_ISO8601:date-time},%{DATA:client-ip},%{DATA:client-hostname},%{DATA:Server-ip},%{DATA:server-hostname},%{DATA:source-context},%{DATA:connector-id},%{DATA:source},%{DATA:event-id},%{DATA:internal-message-id},%{DATA:message-id},%{DATA:network-message-id},%{DATA:recipient-address},%{DATA:recipient-status},%{DATA:total-bytes},%{DATA:recipient-count},%{DATA:related-recipient-address},%{DATA:reference},%{DATA:message-subject},%{DATA:sender-address},%{DATA:return-path},%{DATA:message-info},%{DATA:directionality},%{DATA:tenant-id},%{DATA:original-client-ip},%{DATA:original-server-ip},%{DATA:custom-data},%{DATA:transport-traffic-type}"] } }, { "remove" : { "field" : "message" } }], "on_failure": [ { "set" : { "field" : "error", "value" : " - Error processing message - " } } ] } ``` Log Message to parse ```auto a#Software: Microsoft Exchange Server #Version: 15.01.0225.037 #Log-type: Message Tracking Log #Date: 2020-07-14T17:13:37.768Z #Fields: date-time,client-ip,client-hostname,server-ip,server-hostname,source-context,connector-id,source,event-id,internal-message-id,message-id,network-message-id,recipient-address,recipient-status,total-bytes,recipient-count,related-recipient-address,reference,message-subject,sender-address,return-path,message-info,directionality,tenant-id,original-client-ip,original-server-ip,custom-data,transport-traffic-type 2020-07-14T17:13:38.009Z,10.0.0.2,QQQ0c.Random.CU,10.0.0.3,QQQ0c,08D9463CB6EE1755;2020-07-14T17:13:37.574Z;0,QQQ0c\Default QQQ0c,SMTP,RECEIVE,4449933333,,1c62d03e-54c8-43e7-e5bc-05d946eab7f5,,,10429,1,,,0000003d-0000-0000-0000-0000b70d6900-MapiSubmitLAMProbe,HealthMailbox035ff7daed444848be3484e733389269@AAA.Random.cu,HealthMailbox035ff7daedec3848be3487354c583569@Random.cu.qw,0cI: ,Originating,,::1,10.0.0.3,S:FirstForestHop=QQQc.RandomCU;S:FromEntity=Hosted;S:ProxiedClientIPAddress=10.0.03;S:ProxiedClientHostname=QQQc.Random.CU;S:ProxyHop1=QQQc.Random.CU(10.0.0.3);S:DeliveryPriority=Normal;S:AccountForest=Random.cu;S:IsProbe=true;S:PersistProbeTrace=False,Email ``` filebeat config ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/3/c3d850be582e8fef617442d4f18392ed79bae3cb.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/2/42ee559dae46610fb7a9caf815ddc7c1b0f2f25c.png) ---
**Author:** ![Felix\_Roessel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felix_roessel/32/41623_2.png) [@Felix\_Roessel](https://discuss.elastic.co/u/Felix_Roessel)\ **Post date:** [August 9, 2021, 4:33pm UTC](https://discuss.elastic.co/t/having-problem-parsing-message-via-ingest-pipeline/280827/2 "2021-08-09T16:33:26Z")
Wouldn't it be easier using the CSV processor directly? > **[CSV processor | Elasticsearch Guide \[master\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/csv-processor.html)** ---
**Author:** ![Alaxwora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alaxwora/32/76976_2.png) [@Alaxwora](https://discuss.elastic.co/u/Alaxwora)\ **Post date:** [August 10, 2021, 7:14am UTC](https://discuss.elastic.co/t/having-problem-parsing-message-via-ingest-pipeline/280827/3 "2021-08-10T07:14:11Z")
in this case, Yes I used dissect, CSV and Grok and CSV was the best choice but my main problem the documents are not parsed even after adding Pipeline Id in the config file ---
**Author:** ![Felix\_Roessel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felix_roessel/32/41623_2.png) [@Felix\_Roessel](https://discuss.elastic.co/u/Felix_Roessel)\ **Post date:** [August 10, 2021, 11:29am UTC](https://discuss.elastic.co/t/having-problem-parsing-message-via-ingest-pipeline/280827/4 "2021-08-10T11:29:49Z")
Your Screenshot has truncated the error message. I can just see that it fails. Usually it also says why it fails. Can you add the full error message so what we can better read it? Also your example log starts with `a` before the # sign.. If this is not a typo then you need to exclude this line also. ---
**Author:** ![Alaxwora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alaxwora/32/76976_2.png) [@Alaxwora](https://discuss.elastic.co/u/Alaxwora)\ **Post date:** [August 10, 2021, 12:01pm UTC](https://discuss.elastic.co/t/having-problem-parsing-message-via-ingest-pipeline/280827/5 "2021-08-10T12:01:02Z")
That was the full error the rest lines in PowerShell showing the same error messages for the new lines in log file ![59c11c88f3f4e91cfbd6927c0d1e16ad1f9fcad7](https://us1.discourse-cdn.com/elastic/original/3X/e/0/e0d926afbd20b854c40a725417c52be4c1cf3c0c.png) ---
**Author:** ![Felix\_Roessel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felix_roessel/32/41623_2.png) [@Felix\_Roessel](https://discuss.elastic.co/u/Felix_Roessel)\ **Post date:** [August 10, 2021, 2:28pm UTC](https://discuss.elastic.co/t/having-problem-parsing-message-via-ingest-pipeline/280827/6 "2021-08-10T14:28:58Z")
This Error says that you want to ingest a document that has a conflicting data structure / mapping for field source. The existing field has a concrete value while your new document wants to ingest an object (multiple fields) . If you dont need the existing data the easiest way is to delete the index. If you need the existing data you can switch the index or change the target field for your parsing. ---
**Author:** ![Alaxwora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alaxwora/32/76976_2.png) [@Alaxwora](https://discuss.elastic.co/u/Alaxwora)\ **Post date:** [August 10, 2021, 2:51pm UTC](https://discuss.elastic.co/t/having-problem-parsing-message-via-ingest-pipeline/280827/7 "2021-08-10T14:51:17Z")
I changed the index "source" to another name and it worked thanks \<3 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 7, 2021, 2:52pm UTC](https://discuss.elastic.co/t/having-problem-parsing-message-via-ingest-pipeline/280827/8 "2021-09-07T14:52:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.