Trouble with escaped fields in nested JSON... Causing LogStash Json ParserError: Unrecognized token

Elastic Stack Logstash
1

I'm having trouble parsing nested JSON from events. I'm concerned I'm not dealing with the escaped quotation marks properly, or missing some interim step.

Original entry as written to the log
{"timeMillis":1524260622838,"thread":"https-jsse-nio-8004-exec-10","level":"INFO","loggerName":"xor.bcmc.flarecloud.utils.http.filter.ReqResFilter","message":"Response sent: {\"response\":{\"status-code\":200,\"headers\":[{\"content-encoding\":\"UTF-8\",\"content-type\":\"application/json;charset=UTF-8\"},{},{\"X-Content-Type-Options\":\"nosniff\"},{\"X-XSS-Protection\":\"1; mode=block\"},{\"Cache-Control\":\"no-cache, no-store, max-age=0, must-revalidate\"},{\"Pragma\":\"no-cache\"},{\"Expires\":\"0\"},{\"Strict-Transport-Security\":\"max-age=31536000 ; includeSubDomains\"},{\"X-Frame-Options\":\"DENY\"},{\"Set-Cookie\":\"JSESSIONID=74A1B323C877C840F3E00232697F9BF1; Path=/; Secure; HttpOnly\"},{\"X-Application-Context\":\"users:8004\"},{\"Content-Type\":\"application/json;charset=UTF-8\"},{\"Transfer-Encoding\":\"chunked\"},{\"Date\":\"Fri, 20 Apr 2018 21:43:42 GMT\"}],\"message-body\":{\"_id\":\"0\",\"username\":\"admin\",\"password\":\"$2a$10$HfDlF5ylNQAJSAe/bZer5e0Ync5QK/Qo8u/sFWJd2Us014FKBWmH2\",\"roles\":[{\"role\":\"ROLE_ADMIN\"}],\"tags\":[],\"certificate\":{\"id\":\"5ada5170bf15d3269b8052d7\",\"issuer_hash\":50917989,\"serial_number\":13798258158239537265,\"not_before\":\"2017-09-12T20:00:13Z\",\"not_after\":\"2018-09-12T20:00:13Z\"}}}}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{"x-transaction-id":"802ec34b-fbc3-4728-b0ad-fd1c582f05c8"},"threadId":43,"threadPriority":5}

Filter config
filter {

    json {
        source => "message"
        target => "jlog"
      }

    grok {
        match => { "[jlog]" => "(Response sent:) %{GREEDYDATA:object}" }
      }  

    mutate {
      gsub => ["object","[\\]",""]
    }  

      json {
        source => "[object]"
        target => "jobject"
      }


      mutate {
        add_field => {
          "status-code" => "%{[object][status-code]}"
        }
      }
}

Error displayed in logstash-plain.log

[2018-04-20T21:44:33,601][WARN ][logstash.filters.json    ] Error parsing json {:source=>"[object]", :raw=>"{\"response\":{\"status-code\":200,\"headers\":[{\"content-encoding\":\"UTF-8\",\"content-type\":\"application/json;charset=UTF-8\"},{},{\"X-Content-Type-Options\":\"nosniff\"},{\"X-XSS-Protection\":\"1; mode=block\"},{\"Cache-Control\":\"no-cache, no-store, max-age=0, must-revalidate\"},{\"Pragma\":\"no-cache\"},{\"Expires\":\"0\"},{\"Strict-Transport-Security\":\"max-age=31536000 ; includeSubDomains\"},{\"X-Frame-Options\":\"DENY\"},{\"Set-Cookie\":\"JSESSIONID=74A1B323C877C840F3E00232697F9BF1; Path=/; Secure; HttpOnly\"},{\"X-Application-Context\":\"users:8004\"},{\"Content-Type\":\"application/json;charset=UTF-8\"},{\"Transfer-Encoding\":\"chunked\"},{\"Date\":\"Fri, 20 Apr 2018 21:43:42 GMT\"}],\"message-body\":{\"_id\":\"0\",\"username\":\"admin\",\"password\":\"$2a$10$HfDlF5ylNQAJSAe/bZer5e0Ync5QK/Qo8u/sFWJd2Us014FKBWmH2\",\"roles\":[{\"role\":\"ROLE_ADMIN\"}],\"tags\":[],\"certificate\":{\"id\":\"5ada5170bf15d3269b8052d7\",\"issuer_hash\":50917989,\"serial_number\":13798258158239537265,\"not_before\":\"2017-09-12T20:00:13Z\",\"not_after\":\"2018-09-12T20:00:13Z\"}}}}\", \"threadId\"=>43}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'threadId': was expecting 'null', 'true', 'false' or NaN
 at [Source: (byte[])"{"response":{"status-code":200,"headers":[{"content-encoding":"UTF-8","content-type":"application/json;charset=UTF-8"},{},{"X-Content-Type-Options":"nosniff"},{"X-XSS-Protection":"1; mode=block"},{"Cache-Control":"no-cache, no-store, max-age=0, must-revalidate"},{"Pragma":"no-cache"},{"Expires":"0"},{"Strict-Transport-Security":"max-age=31536000 ; includeSubDomains"},{"X-Frame-Options":"DENY"},{"Set-Cookie":"JSESSIONID=74A1B323C877C840F3E00232697F9BF1; Path=/; Secure; HttpOnly"},{"X-Application-"[truncated 507 bytes]; line: 1, column: 1003]>}
2

Multiple problems:

  • The JSON string you want to parse is in the [jlog][message] field, not [jlog].
  • Don't remove any backslashes with gsub.
3

To be clear, you're saying in the grok filter to use [jlog][message]?

Like so?

filter {
    
    json {
        source => "message"
        target => "jlog"
      }

    grok {
        match => { "[jlog][message]" => "(Response sent:) %{GREEDYDATA:object}" }
      }  

      json {
        source => "[object]"
        target => "jobject"
      }


      mutate {
        add_field => {
          "status-code" => "%{[object][status-code]}"
        }
      }
}
4

Yes, this should work. Why not try it out? Build your set of filters gradually and you'll see what works and what doesn't.

Speaking of things that don't work, your final mutate attempts to address a subfield from the string field. object. The parsed JSON string is in the jobject field, and the status-code subfield is nested even further down so what you're looking for is %{[jobject][response][status-code]}.

5

That works beautifully! Thank you very much!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.