# Remove Special character from message before sending to json filter **URL:** \ **Category:** Logstash\ **Created:** [June 29, 2022, 9:04am UTC](https://discuss.elastic.co/t/remove-special-character-from-message-before-sending-to-json-filter/308434 "2022-06-29T09:04:16Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Basel](https://avatars.discourse-cdn.com/v4/letter/b/ac91a4/32.png) [@Basel](https://discuss.elastic.co/u/Basel)\ **Post date:** [June 29, 2022, 9:04am UTC](https://discuss.elastic.co/t/remove-special-character-from-message-before-sending-to-json-filter/308434/1 "2022-06-29T09:04:16Z")
Hi, we are using ELK with Apigee to send the transaction logs, the logstash configuration is as recommended by the community as below: ```auto input { tcp { port => 8080 type => syslog } } filter { mutate { gsub => ["message", "[\u0000]", ""] } grok { match => {"message" => "<%{NUMBER:priority_index}>%{DATESTAMP_OTHER:apigeeTimestamp}%{LOGLEVEL}: %{GREEDYDATA:apigeeMessage}"} remove_field => ["message"] } json { source => "apigeeMessage" remove_field => ["apigeeMessage"] } } output { #added below line to see how the message is outputed/parsed with logstash filer, you can remove below line (only stdout { codec => rubydebug }), if necessary. stdout { codec => rubydebug } elasticsearch { hosts => ["IP1", "IP2", "IP3"] index => "apigee-%{+YYYY.MM}" user => "elastic" password => "*" } } ``` the real wanted json message contain a field called: error.message ```auto "error":{ "isError":"true", "message":"Execution of JS-ValidateRequiredHeadersAndQueryParams failed with error: "Javascript runtime" error: ", "errorCode":"500", "errorPhrase":"Internal Server Error", "transportMessage":"com.apigee.messaging.adaptors.http.message.HttpResponseMessage@73f118a9", "errorState":"PROXY_REQ_FLOW", "isPolicyError":"1", "isTargetError":"0", "policyErrorPolicyName":"JS-ValidateRequiredHeadersAndQueryParams", "policyErrorFlowName":"contracts", "error":"com.apigee.flow.message.MessageImpl@6d128cc9", "content":"" } } ``` * * * the issue is when the json filter receive the message it gives error \_jsonparsingerror because the error.message contain extra (" ") and the filter cannot pars the message is there any way to clear the filed error.message from any special character without damaging the json structure, i tried the below but it did not work : gsub =\> ["[apigeeMessage][error][message]", """, "" ] second issue: the error.content will contain another json message is there a way to manage it?. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 29, 2022, 5:59pm UTC](https://discuss.elastic.co/t/remove-special-character-from-message-before-sending-to-json-filter/308434/2 "2022-06-29T17:59:12Z")
It can be done, although this might be rather fragile... ``` grok { match => { "message" => '"message":"(?([-"[:punct]\w\s]+))(?=",)' } } mutate { gsub => ["internalMessage", '"', '\"'] } mutate { gsub => ["message", '(?="message":")([-"[:punct]\w\s]+)",', '"message":"%{internalMessage}",' ] } json { source => "message" } ``` will get you ``` "error" => { "transportMessage" => "com.apigee.messaging.adaptors.http.message.HttpResponseMessage@73f118a9", ... "message" => "Execution of JS-ValidateRequiredHeadersAndQueryParams failed with error: \"Javascript runtime\" error: ", "content" => "" }, ``` I am absolutely baffled as to why ``` grok { match => { "message" => '(?="message":")(?([-"[:punct]\w\s]+))(?!,)' } } ``` results in ``` "internalMessage" => "\"message\":\"Execution of JS-ValidateRequiredHeadersAndQueryParams failed with error: \"Javascript runtime\" error: " ``` Why is the lookahead included in the resulting match? ---
**Author:** ![Basel](https://avatars.discourse-cdn.com/v4/letter/b/ac91a4/32.png) [@Basel](https://discuss.elastic.co/u/Basel)\ **Post date:** [June 30, 2022, 7:48am UTC](https://discuss.elastic.co/t/remove-special-character-from-message-before-sending-to-json-filter/308434/3 "2022-06-30T07:48:23Z")
thanks for your input, should i remove old filters? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 30, 2022, 4:45pm UTC](https://discuss.elastic.co/t/remove-special-character-from-message-before-sending-to-json-filter/308434/4 "2022-06-30T16:45:43Z")
Not entirely, it looks like you are getting a syslog message, so you need the grok that processes the prefix. The filters I mentioned will process a field that contains ``` { "error": { "isError":"true", "message":"Execution of ... } } ``` in your case what I called "message" is probably your [apigeeMessage] ---
**Author:** ![Basel](https://avatars.discourse-cdn.com/v4/letter/b/ac91a4/32.png) [@Basel](https://discuss.elastic.co/u/Basel)\ **Post date:** [July 5, 2022, 8:10am UTC](https://discuss.elastic.co/t/remove-special-character-from-message-before-sending-to-json-filter/308434/5 "2022-07-05T08:10:46Z")
sorry for the late, i tried the below filter but i amgetting "\_grokparsefailure" and "\_jsonparsefailure" ```auto filter { mutate { gsub => ["message", "[\u0000]", ""] remove_field => ["timestamp", "host", "facility_label", "severity_label", "severity", "facility", "priority"] } grok { match => {"message" => "<%{NUMBER:priority_index}>%{DATESTAMP_OTHER:apigeeTimestamp}%{LOGLEVEL}: %{GREEDYDATA:apigeeMessage}"} } grok { match => { "apigeeMessage" => '"apigeeMessage":"(?([-"[:punct]\w\s]+))(?=",)' } } mutate { gsub => ["internalMessage", '"', '\"'] } mutate { gsub => ["apigeeMessage", '(?="apigeeMessage":")([-"[:punct]\w\s]+)",', '"apigeeMessage":"%{internalMessage}",' ] } json { source => "apigeeMessage" } } ``` and below is a sample of apigeeMessage : `{\"RqUID\":\"432dee33-ca09-4e96-a3f7-f07d5a72afd4\",\"requestReceivedTs\":\"2022-6-30T10:30:41.1656574241016\",\"requestSentTs\":\"1970-01-01T00:00:00.000\",\"ResponsePayload\":\"\",\"RequestPayload\":\"\",\"responseReceivedTs\":\"1970-01-01T00:00:00.000\",\"responseSentTs\":\"1969-12-31T23:59:59.999\",\"systemTimeStamp\":\"2022-6-30T10:30:42.57Z\",\"messageId\":\"server-84361-15600-1\",\"statusCode\":\"500\",\"RequestHeaders\":\"\",\"ResponseHeaders\":\"\",\"contentType\":\"\",\"clientID\":\"hCOcfA1VzJYYZD46gRoru9VeDk\",\"contentLength\":\"0\",\"uri\":\"/banks/v1\",\"verb\":\"GET\",\"meta\":{\"organization\":\"test\",\"product\":\"TestingProduct\",\"application\":\"TestingApp\",\"environment\":\"preprod\",\"virtualhost\":\"internal-LB\"},\"client\":{\"IP\":\"1.1.1.1\",\"host\":\"1.1.1.1\"},\"proxy\":{\"apiName\":\"Proxy-v1\",\"name\":\"default\",\"revision\":\"6\",\"url\":\"http://1.1.1.1"},\"target\":{\"host\":\"\",\"basepath\":\"/path\",\"targetBasePath\":\"\",\"name\":\"\",\"statusCode\":\"\",\"contentLength\":\"\",\"URL\":\"\"},\"error\":{\"isError\":\"true\",\"message\":\"Execution of JS-ValidateRequiredHeadersAndQueryParams failed with error: Javascript runtime error: \"TypeError: Cannot call method \"split\" of null. (ValidateRequiredHeadersAndQueryParams.js:48)\"\",\"errorCode\":\"500\",\"errorPhrase\":\"Internal Server Error\",\"transportMessage\":\"com.apigee.messaging.adaptors.http.message.HttpResponseMessage@6015e8e0\",\"errorState\":\"PROXY_REQ_FLOW\",\"isPolicyError\":\"1\",\"isTargetError\":\"0\",\"policyErrorPolicyName\":\"JS-ValidateRequiredHeadersAndQueryParams\",\"policyErrorFlowName\":\"contracts\",\"error\":\"com.apigee.flow.message.MessageImpl@38311235\",\"content\":\"\"}}` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 2, 2022, 8:11am UTC](https://discuss.elastic.co/t/remove-special-character-from-message-before-sending-to-json-filter/308434/6 "2022-08-02T08:11:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.