# Ingest pipeline created for filebeat log using grok pattern,but unable to run in dev tools **URL:** \ **Category:** Beats\ **Tags:** filebeat, ingest-pipeline\ **Created:** [March 14, 2023, 12:48pm UTC](https://discuss.elastic.co/t/ingest-pipeline-created-for-filebeat-log-using-grok-pattern-but-unable-to-run-in-dev-tools/327660 "2023-03-14T12:48:22Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![PRASHANT\_MEHTA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prashant_mehta/32/101764_2.png) [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Post date:** [March 14, 2023, 12:48pm UTC](https://discuss.elastic.co/t/ingest-pipeline-created-for-filebeat-log-using-grok-pattern-but-unable-to-run-in-dev-tools/327660/1 "2023-03-14T12:48:22Z")
Hello All, I'm successfully getting output of my log pattern using grok pattern,but when trying to create log ingest pipeline that can be used in template later,I'm getting exception while running the ingest pipeline in dev tools. Kindly suggest,how to rectify ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/9/69837a85798632827593a16c6da2e275f699dafd.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/0/3074243b9abc4df806ea71983b17cd64d58c4312.png) ```auto abcd02appl002_MONITORING_USECASES|abcnctd02appl002|MONITORING_USECASES|23653|/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log|2023-03-13 15:40:31|2023-03-13 15:40:34|0h 0min 3sec|2023-03-13 16:00:00|0.0|0.1|188360|33204|2796|perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/configuration.properties|/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl ``` ```auto %{WORD:UNIQUEID}\|%{WORD:SERVERNAME}\|%{WORD:USECASENAME}\|%{NUMBER:PID}\|%{UNIXPATH:LOGPATH}\|%{TIMESTAMP_ISO8601:STARTTIME}\|%{TIMESTAMP_ISO8601:ENDTIME}\|%{DATA:DURATION}\|%{TIMESTAMP_ISO8601:NEXTEVENT}\|%{NUMBER:CPUUSAGE}\|%{NUMBER:MEMORYUSAGE}\|%{NUMBER:VIRTUALMEMORY}\|%{NUMBER:RESIDENTMEMORY}\|%{NUMBER:SHAREDMEMORY}\|%{GREEDYDATA:COMMAND}\|%{GREEDYDATA:PERLFILE} ``` output: ```auto { "PERLFILE": "/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl", "USECASENAME": "MONITORING_USECASES", "ENDTIME": "2023-03-13 15:40:34", "PID": "23653", "RESIDENTMEMORY": "33204", "SHAREDMEMORY": "2796", "SERVERNAME": "abcnctd02appl002", "STARTTIME": "2023-03-13 15:40:31", "UNIQUEID": "abcd02appl002_MONITORING_USECASES", "VIRTUALMEMORY": "188360", "CPUUSAGE": "0.0", "MEMORYUSAGE": "0.1", "COMMAND": "perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/globalconfiguration.properties", "NEXTEVENT": "2023-03-13 16:00:00", "DURATION": "0h 0min 3sec", "LOGPATH": "/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log" } ``` ```auto PUT _ingest/pipeline/mis-usecases-ingest-pipeline { "description": "Ingest Pipeline for incoming log documents from Filebeat agents ", "processors": [ { "grok": { "field": "message", "patterns": [ "%{WORD:UNIQUEID}\|%{WORD:SERVERNAME}\|%{WORD:USECASENAME}\|%{NUMBER:PID}\|%{UNIXPATH:LOGPATH}\|%{TIMESTAMP_ISO8601:STARTTIME}\|%{TIMESTAMP_ISO8601:ENDTIME}\|%{DATA:DURATION}\|%{TIMESTAMP_ISO8601:NEXTEVENT}\|%{NUMBER:CPUUSAGE}\|%{NUMBER:MEMORYUSAGE}\|%{NUMBER:VIRTUALMEMORY}\|%{NUMBER:RESIDENTMEMORY}\|%{NUMBER:SHAREDMEMORY}\|%{GREEDYDATA:COMMAND}\|%{GREEDYDATA:PERLFILE}" ], "on_failure": [ { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } } ] } } ] } ``` Exception:Unable to understand ```auto { "error" : { "root_cause" : [ { "type" : "parse_exception", "reason" : "Failed to parse content to map" } ], "type" : "parse_exception", "reason" : "Failed to parse content to map", "caused_by" : { "type" : "json_parse_exception", "reason" : "Unrecognized character escape '|' (code 124)\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 8, column: 30]" } }, "status" : 400 } ``` How can I create proper log ingest pileline? Thanx ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 14, 2023, 12:56pm UTC](https://discuss.elastic.co/t/ingest-pipeline-created-for-filebeat-log-using-grok-pattern-but-unable-to-run-in-dev-tools/327660/2 "2023-03-14T12:56:55Z")
I'm not sure you need to escape the `|`, have you tried without escaping it? Also, if your message looks like it, it would be better to use the `dissect` processor or you can even use the `csv` processor setting the `|` as a separator. ---
**Author:** ![PRASHANT\_MEHTA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prashant_mehta/32/101764_2.png) [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Post date:** [March 14, 2023, 2:21pm UTC](https://discuss.elastic.co/t/ingest-pipeline-created-for-filebeat-log-using-grok-pattern-but-unable-to-run-in-dev-tools/327660/3 "2023-03-14T14:21:32Z")
Hello @leandrojmp , By escaping the **|** I don't get any data.I'm still not sure why In kibana dev tools I'm getting correct output with grok pattern created but while executing my ingest pipeline using PUT ,I'm getting the error. I came to this solution after reading this statement in net: Unrecognized character escape '|'" is most likely caused by the pipe character '|' in your grok pattern. The pipe character is a special character in regular expressions, and needs to be escaped with a backslash. after adding \ I'm able to parse data in GROK Debugger in devtools,but in console getting error. I'm not sure how disscet or csv would help here,would explore if that works. Thanx ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 14, 2023, 3:33pm UTC](https://discuss.elastic.co/t/ingest-pipeline-created-for-filebeat-log-using-grok-pattern-but-unable-to-run-in-dev-tools/327660/4 "2023-03-14T15:33:55Z")
@PRASHANT_MEHTA Try this note the triple quotes on the pattern... ```auto PUT _ingest/pipeline/mis-usecases-ingest-pipeline { "description": "Ingest Pipeline for incoming log documents from Filebeat agents ", "processors": [ { "grok": { "field": "message", "patterns": [ """%{WORD:UNIQUEID}\|%{WORD:SERVERNAME}\|%{WORD:USECASENAME}\|%{NUMBER:PID}\|%{UNIXPATH:LOGPATH}\|%{TIMESTAMP_ISO8601:STARTTIME}\|%{TIMESTAMP_ISO8601:ENDTIME}\|%{DATA:DURATION}\|%{TIMESTAMP_ISO8601:NEXTEVENT}\|%{NUMBER:CPUUSAGE}\|%{NUMBER:MEMORYUSAGE}\|%{NUMBER:VIRTUALMEMORY}\|%{NUMBER:RESIDENTMEMORY}\|%{NUMBER:SHAREDMEMORY}\|%{GREEDYDATA:COMMAND}\|%{GREEDYDATA:PERLFILE}""" ], "on_failure": [ { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } } ] } } ] } ``` ```auto POST _ingest/pipeline/mis-usecases-ingest-pipeline/_simulate { "docs": [ { "_source" : { "message" : "abcd02appl002_MONITORING_USECASES|abcnctd02appl002|MONITORING_USECASES|23653|/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log|2023-03-13 15:40:31|2023-03-13 15:40:34|0h 0min 3sec|2023-03-13 16:00:00|0.0|0.1|188360|33204|2796|perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/configuration.properties|/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl" } } ] } ``` Results ```auto { "docs": [ { "doc": { "_index": "_index", "_id": "_id", "_version": "-3", "_source": { "PERLFILE": "/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl", "USECASENAME": "MONITORING_USECASES", "ENDTIME": "2023-03-13 15:40:34", "PID": "23653", "RESIDENTMEMORY": "33204", "message": "abcd02appl002_MONITORING_USECASES|abcnctd02appl002|MONITORING_USECASES|23653|/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log|2023-03-13 15:40:31|2023-03-13 15:40:34|0h 0min 3sec|2023-03-13 16:00:00|0.0|0.1|188360|33204|2796|perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/configuration.properties|/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl", "SHAREDMEMORY": "2796", "SERVERNAME": "abcnctd02appl002", "STARTTIME": "2023-03-13 15:40:31", "UNIQUEID": "abcd02appl002_MONITORING_USECASES", "VIRTUALMEMORY": "188360", "CPUUSAGE": "0.0", "MEMORYUSAGE": "0.1", "COMMAND": "perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/configuration.properties", "NEXTEVENT": "2023-03-13 16:00:00", "DURATION": "0h 0min 3sec", "LOGPATH": "/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log" }, "_ingest": { "timestamp": "2023-03-14T15:33:16.145176498Z" } } } ] } ``` ---
**Author:** ![PRASHANT\_MEHTA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prashant_mehta/32/101764_2.png) [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Post date:** [March 15, 2023, 11:43am UTC](https://discuss.elastic.co/t/ingest-pipeline-created-for-filebeat-log-using-grok-pattern-but-unable-to-run-in-dev-tools/327660/5 "2023-03-15T11:43:12Z")
Hello @stephenb , Thanx for your time to look into this.This works now. `""" """` : this really helped. Thanx ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 12, 2023, 1:43pm UTC](https://discuss.elastic.co/t/ingest-pipeline-created-for-filebeat-log-using-grok-pattern-but-unable-to-run-in-dev-tools/327660/6 "2023-04-12T13:43:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.