Error parsing JSON content from Http

Elastic Stack Logstash
1

Hello to everybody.
I am really noob on elastic stack and I am trying to import some json information with the HTTP input to Elasticsarch, but I recive a parse error:
exception=>#<LogStash::Json::ParserError: Unrecognized token 'json': was expecting ('true', 'false' or 'null')

This is how i recive the information on my elasticsearch:

"_type" : "_doc",
        "_id" : "vCS44WsBBF5AknIKMFMZ",
        "_score" : 1.0,
        "_source" : {
          "message" : """json={"id":"136030541","domain":"sdl","priority":"5.0","context":"my.alert.sdl.Error500","srcIp":"","dstIp":"","srcPort":"","dstPort":"","category":"my.context","protocol":"","username":"","application":"","status":"4","creation_date":"Thu Jul 11 15:47:52 UTC 2019","ack_status_date":null,"digest":"e738f645464be99ff223b504ec4a213d92ce8c6f","scaled":"false","srcHost":"","dstHost":"","extraData":"{\"clientIpAddress\":\"null\",\"timeTaken\":\"463\",\"protocol\":\"HTTP+1.1\",\"bytesTransferred\":\"677\",\"method\":\"POST\",\"cookie\":\"gaqfse5dpcm690jdh5ho1f00o2%3A-\",\"userAgent\":\"Mozilla%2F4.0+%28compatible%3B+MSIE+6.0%3B+Windows+NT+5.1%29\",\"referralUri\":\"http%3A%2F%2Fwww.logcasts.com%2Fcart.do%3Faction%3Dview%26itemId%3DLOG-56%26product_id%3D235-40ER0-J3308%26JSESSIONID%3DSD3SL2FF4ADFF4\",\"uri\":\"%2Fcategory.screen%3Fcategory_id%3DFURNITURE%26JSESSIONID%3DSD3SL2FF4ADFF4\",\"eventdate\":\"2019-07-11+11%3A56%3A57.04\",\"timestamp\":\"11%2FJul%2F2019%3A11%3A56%3A53+%2B0000\",\"statusCode\":\"500\"}","engine":"pilot.my.alert.samps.Error500","textMessage":"Error 500","shortMessage":"Error 500","alertname":"Error500","categoryname":"My Alerts","subcategoryname":"Personal"}""",
          "headers" : {
            "request_path" : "/",
            "request_method" : "POST",
            "http_accept" : null,
            "content_length" : "1219",
            "connection" : "Keep-Alive",
            "http_user_agent" : "Apache-HttpClient/4.5.2 (Java/1.8.0_191)",
            "http_host" : "XXXXXXXXXXXXXXX:8080",
            "http_version" : "HTTP/1.1",
            "content_type" : "application/x-www-form-urlencoded"
          },
          "host" : "XXXXXXXXXXXXX",
          "@version" : "1",
          "@timestamp" : "2019-07-11T15:47:52.855Z"
        }

I want to parse the message field with this configuration:

filter {
  json {
    source => "message"
 }
}

Could anyone help me with this? I think that could be the format of the message field, but i dont know how i could alter it.

Thanks to all.

2

Yes, the problem is that the message field has other stuff prefixed the JSON. Please edit your post, select the configuration, and click on </> in the toolbar above the edit pane. If you look in the preview pane on the right it will start appearing

like this

Then do the same for document data. At the moment the browser is removing all the \ characters that are escaped in the message field, so it is too hard to tell what you need to remove.

3

Thanks Badger, post edited.
I must to add that i cant change the way i recibe the data, I have to edit the message content with a logstash filter, I suppose.

4

Something like

    mutate { gsub => [ "message", '^""json=', "", "message", '""$', "" ] }
    json { source => "message" target => "jsondata1" }
    json { source => "[jsondata1][extraData]" target => "jsondata2" }
5

Thanks Badger,
I try it but dont work. This is the console output.

[WARN ] 2019-07-12 06:51:30.307 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"json={\"id\":\"136788731\",\"domain\":\"spb\",\"priority\":\"5.0\",\"context\":\"my.alert.spb.Error500\",\"srcIp\":\"\",\"dstIp\":\"\",\"srcPort\":\"\",\"dstPort\":\"\",\"category\":\"my.context\",\"protocol\":\"\",\"username\":\"\",\"application\":\"\",\"status\":\"4\",\"creation_date\":\"Fri Jul 12 06:51:30 UTC 2019\",\"ack_status_date\":null,\"digest\":\"48b7c6a38d3566dbfa9f4580104056a0351f00ff\",\"scaled\":\"false\",\"srcHost\":\"\",\"dstHost\":\"\",\"extraData\":\"{\\\"clientIpAddress\\\":\\\"null\\\",\\\"timeTaken\\\":\\\"227\\\",\\\"protocol\\\":\\\"HTTP+1.1\\\",\\\"bytesTransferred\\\":\\\"3317\\\",\\\"method\\\":\\\"GET\\\",\\\"cookie\\\":\\\"3djv1l0ebi7cmsai1131pf2a65%3A-\\\",\\\"userAgent\\\":\\\"Mozilla%2F4.0+%28compatible%3B+MSIE+6.0%3B+Windows+NT+5.1%29\\\",\\\"referralUri\\\":\\\"http%3A%2F%2Fwww.google.com%2Fproduct.screen%3Fproduct_id%3DL98-72BOK-SKD00%26JSESSIONID%3DSD3SL9FF8ADFF9\\\",\\\"uri\\\":\\\"%2Fcategory.screen%3Fcategory_id%3DFURNITURE%26JSESSIONID%3DSD3SL9FF8ADFF9\\\",\\\"eventdate\\\":\\\"2019-07-12+06%3A45%3A18.141\\\",\\\"timestamp\\\":\\\"12%2FJul%2F2019%3A06%3A44%3A56+%2B0000\\\",\\\"statusCode\\\":\\\"500\\\"}\",\"engine\":\"pilot.my.alert.spb.Error500\",\"textMessage\":\"Error 500\",\"shortMessage\":\"Error 500\",\"alertname\":\"Error500\",\"categoryname\":\"My Alerts\",\"subcategoryname\":\"Personal\"}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'json': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"json={"id":"136788731","domain":"spb","priority":"5.0","context":"my.alert.spb.Error500","srcIp":"","dstIp":"","srcPort":"","dstPort":"","category":"my.context","protocol":"","username":"","application":"","status":"4","creation_date":"Fri Jul 12 06:51:30 UTC 2019","ack_status_date":null,"digest":"48b7c6a38d3566dbfa9f4580104056a0351f00ff","scaled":"false","srcHost":"","dstHost":"","extraData":"{\"clientIpAddress\":\"null\",\"timeTaken\":\"227\",\"protocol\":\"HTTP+1.1\",\"byt"[truncated 691 bytes]; line: 1, column: 6]>}

and this is how i recive on elasticsearch:

#  _score	 - 
  	t  _type	_doc
   	t  headers.connection	Keep-Alive
   	t  headers.content_length	1187
 	?  headers.content_type	  application/x-www-form-urlencoded
   	t  headers.http_accept	 - 
   	t  headers.http_host	XXXXXXXXXXXXXXX
   	t  headers.http_user_agent	Apache-HttpClient/4.5.2 (Java/1.8.0_191)
   	t  headers.http_version	HTTP/1.1
   	t  headers.request_method	POST
   	t  headers.request_path	/
   	t  host	XXXXXXXXXXXXXXXXX
   	t  message	json={"id":"136788651","domain":"spb","priority":"5.0","context":"my.alert.spb.Error500","srcIp":"","dstIp":"","srcPort":"","dstPort":"","category":"my.context","protocol":"","username":"","application":"","status":"4","creation_date":"Fri Jul 12 06:49:35 UTC 2019","ack_status_date":null,"digest":"0c86affb0499bac0f1413ab2a480c5fe93a605a7","scaled":"false","srcHost":"","dstHost":"","extraData":"{\"clientIpAddress\":\"%2F70.44.164.7\",\"timeTaken\":\"370\",\"protocol\":\"HTTP+1.1\",\"bytesTransferred\":\"1066\",\"method\":\"GET\",\"cookie\":\"3djv1l0ebi7cmsai1131pf2a65%3A-\",\"userAgent\":\"Opera%2F9.01+%28Windows+NT+5.1%3B+U%3B+en%29\",\"referralUri\":\"http%3A%2F%2Fwww.logcasts.com%2Foldlink%3Fitem_id%3DLOG-77%26port%3D161%26JSESSIONID%3DSD8SL10FF6ADFF8\",\"uri\":\"%2Fproduct.screen%3Fproduct_id%3DL98-72BOK-SKD00%26JSESSIONID%3DSD8SL10FF6ADFF8\",\"eventdate\":\"2019-07-12+06%3A44%3A06.039\",\"timestamp\":\"12%2FJul%2F2019%3A06%3A43%3A44+%2B0000\",\"statusCode\":\"500\"}","engine":"pilot.my.alert.spb.Error500","textMessage":"Error 500","shortMessage":"Error 500","alertname":"Error500","categoryname":"My Alerts","subcategoryname":"Personal"}
 	?  tags	  _jsonparsefailure

I am reading documentation about mutate filter and gsub... Thanks again.

I think that i only need to remove de first "json=" text and all"\". This way i have a valid json object but with anoter json object inside, "ExtraData", witch dont parse correctly because if i remove all the "" the field look like this:

"extraData":"{"clientIpAddress":"null","timeTaken":"227","protocol":"HTTP+1.1","bytesTransferred":"3317","method":"GET","cookie":"3djv1l0ebi7cmsai1131pf2a65%3A-","userAgent":"Mozilla%2F4.0+%28compatible%3B+MSIE+6.0%3B+Windows+NT+5.1%29","referralUri":"http%3A%2F%2Fwww.google.com%2Fproduct.screen%3Fproduct_id%3DL98-72BOK-SKD00%26JSESSIONID%3DSD3SL9FF8ADFF9","uri":"%2Fcategory.screen%3Fcategory_id%3DFURNITURE%26JSESSIONID%3DSD3SL9FF8ADFF9","eventdate":"2019-07-12+06%3A45%3A18.141","timestamp":"12%2FJul%2F2019%3A06%3A44%3A56+%2B0000","statusCode":"500"}"

Any idea to a good mutate filter? I still working on it.

Thanks.

6 
mutate { gsub => [ "message", '^json=', "" ] }

That will remove the json= at the beginning.

You do not need to remove the \ in ExtraData, it is just the way logstash displays double quotes within a double quoted string.

1 Like
7

Really thanks you Badger, it was as easy as remove "json="

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.