Remove Special character from message before sending to json filter

Elastic Stack Logstash
1

Hi, we are using ELK with Apigee to send the transaction logs, the logstash configuration is as recommended by the community as below:

input {
    tcp {
        port => 8080
        type => syslog
    }
}
filter {
    mutate {
    gsub => ["message", "[\u0000]", ""]
	
    }
    grok {
        match => {"message" => "<%{NUMBER:priority_index}>%{DATESTAMP_OTHER:apigeeTimestamp}%{LOGLEVEL}: %{GREEDYDATA:apigeeMessage}"}
	remove_field => ["message"]
    }
    
    json {
        source => "apigeeMessage"
        remove_field => ["apigeeMessage"]
	
    }
}
output {
    #added below line to see how the message is outputed/parsed with logstash filer, you can remove below line (only stdout { codec => rubydebug }), if necessary.
        stdout {
                codec => rubydebug
        }
    elasticsearch {
        hosts => ["IP1", "IP2", "IP3"]
        index => "apigee-%{+YYYY.MM}"
        user => "elastic"
        password => "*"

    }
}

the real wanted json message contain a field called: error.message

"error":{
      "isError":"true",
      "message":"Execution of JS-ValidateRequiredHeadersAndQueryParams failed with error: "Javascript runtime" error: ",
      "errorCode":"500",
      "errorPhrase":"Internal Server Error",
      "transportMessage":"com.apigee.messaging.adaptors.http.message.HttpResponseMessage@73f118a9",
      "errorState":"PROXY_REQ_FLOW",
      "isPolicyError":"1",
      "isTargetError":"0",
      "policyErrorPolicyName":"JS-ValidateRequiredHeadersAndQueryParams",
      "policyErrorFlowName":"contracts",
      "error":"com.apigee.flow.message.MessageImpl@6d128cc9",
      "content":""
   }
}

the issue is when the json filter receive the message it gives error _jsonparsingerror because the error.message contain extra (" ") and the filter cannot pars the message is there any way to clear the filed error.message from any special character without damaging the json structure, i tried the below but it did not work :

gsub => ["[apigeeMessage][error][message]", """, "" ]

second issue:
the error.content will contain another json message is there a way to manage it?.

2

It can be done, although this might be rather fragile...

    grok { match => { "message" => '"message":"(?<internalMessage>([-"[:punct]\w\s]+))(?=",)' } }
    mutate { gsub => [ "internalMessage", '"', '\"' ] }
    mutate { gsub => [ "message", '(?="message":")([-"[:punct]\w\s]+)",', '"message":"%{internalMessage}",' ] }
    json { source => "message" }

will get you

          "error" => {
         "transportMessage" => "com.apigee.messaging.adaptors.http.message.HttpResponseMessage@73f118a9",
...
                  "message" => "Execution of JS-ValidateRequiredHeadersAndQueryParams failed with error: \"Javascript runtime\" error: ",
                  "content" => ""
},

I am absolutely baffled as to why

grok { match => { "message" => '(?="message":")(?<internalMessage>([-"[:punct]\w\s]+))(?!,)' } }

results in

"internalMessage" => "\"message\":\"Execution of JS-ValidateRequiredHeadersAndQueryParams failed with error: \"Javascript runtime\" error: "

Why is the lookahead included in the resulting match?

3

thanks for your input, should i remove old filters?

4

Not entirely, it looks like you are getting a syslog message, so you need the grok that processes the prefix. The filters I mentioned will process a field that contains

{ "error": { "isError":"true", "message":"Execution of ... } }

in your case what I called "message" is probably your [apigeeMessage]

5

sorry for the late, i tried the below filter but i amgetting "_grokparsefailure" and "_jsonparsefailure"

filter {
    mutate {
    gsub => ["message", "[\u0000]", ""]
    remove_field => ["timestamp", "host", "facility_label", "severity_label", "severity", "facility", "priority"]
    }
    grok { match => {"message" => "<%{NUMBER:priority_index}>%{DATESTAMP_OTHER:apigeeTimestamp}%{LOGLEVEL}: %{GREEDYDATA:apigeeMessage}"} }
	grok { match => { "apigeeMessage" => '"apigeeMessage":"(?<internalMessage>([-"[:punct]\w\s]+))(?=",)' } }
    mutate { gsub => [ "internalMessage", '"', '\"' ] }
    mutate { gsub => [ "apigeeMessage", '(?="apigeeMessage":")([-"[:punct]\w\s]+)",', '"apigeeMessage":"%{internalMessage}",' ] }
    json { source => "apigeeMessage" }

}

and below is a sample of apigeeMessage :

{\"RqUID\":\"432dee33-ca09-4e96-a3f7-f07d5a72afd4\",\"requestReceivedTs\":\"2022-6-30T10:30:41.1656574241016\",\"requestSentTs\":\"1970-01-01T00:00:00.000\",\"ResponsePayload\":\"\",\"RequestPayload\":\"\",\"responseReceivedTs\":\"1970-01-01T00:00:00.000\",\"responseSentTs\":\"1969-12-31T23:59:59.999\",\"systemTimeStamp\":\"2022-6-30T10:30:42.57Z\",\"messageId\":\"server-84361-15600-1\",\"statusCode\":\"500\",\"RequestHeaders\":\"\",\"ResponseHeaders\":\"\",\"contentType\":\"\",\"clientID\":\"hCOcfA1VzJYYZD46gRoru9VeDk\",\"contentLength\":\"0\",\"uri\":\"/banks/v1\",\"verb\":\"GET\",\"meta\":{\"organization\":\"test\",\"product\":\"TestingProduct\",\"application\":\"TestingApp\",\"environment\":\"preprod\",\"virtualhost\":\"internal-LB\"},\"client\":{\"IP\":\"1.1.1.1\",\"host\":\"1.1.1.1\"},\"proxy\":{\"apiName\":\"Proxy-v1\",\"name\":\"default\",\"revision\":\"6\",\"url\":\"http://1.1.1.1"},\"target\":{\"host\":\"\",\"basepath\":\"/path\",\"targetBasePath\":\"\",\"name\":\"\",\"statusCode\":\"\",\"contentLength\":\"\",\"URL\":\"\"},\"error\":{\"isError\":\"true\",\"message\":\"Execution of JS-ValidateRequiredHeadersAndQueryParams failed with error: Javascript runtime error: \"TypeError: Cannot call method \"split\" of null. (ValidateRequiredHeadersAndQueryParams.js:48)\"\",\"errorCode\":\"500\",\"errorPhrase\":\"Internal Server Error\",\"transportMessage\":\"com.apigee.messaging.adaptors.http.message.HttpResponseMessage@6015e8e0\",\"errorState\":\"PROXY_REQ_FLOW\",\"isPolicyError\":\"1\",\"isTargetError\":\"0\",\"policyErrorPolicyName\":\"JS-ValidateRequiredHeadersAndQueryParams\",\"policyErrorFlowName\":\"contracts\",\"error\":\"com.apigee.flow.message.MessageImpl@38311235\",\"content\":\"\"}}

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.