Logstash 6.2 json parse failure

[Arnav_Sengupta]

Hi,

I am trying to parse the below filebeat log

{  "event": {
"host": {
  "name": "lpcservices-dev-int-alln-75f4d6df98-t9wj2"
},
"beat": {
  "version": "6.4.2",
  "name": "lpcservices-dev-int-alln-75f4d6df98-t9wj2",
  "hostname": "lpcservices-dev-int-alln-75f4d6df98-t9wj2"
},
"source": "/home/jboss/lae-home/app-root/logs/lpcatalogservicesAppIO.log",
"input": {
  "type": "log"
},
"tags": [
  "applicationLogs",
  "beats_input_codec_plain_applied"
],
"message": "2019-06-14 02:25:48 [IOLogger] INFO  {env=STAGE-YS2, serviceType=catalog_ITEM_PUBLISHABLE, transactionId=CAE-TEST-15} - {\"noOfLines\":0,\"request\":\"{\\\"requestHeader\\\":{\\\"transactionID\\\":\\\"CAE-TEST-15\\\",\\\"ccoID\\\":\\\"NETFORMX-XML\\\"},\\\"entries\\\":[{\\\"priceList\\\":\\\"WCH2\\\",\\\"items\\\":[\\\"L-CDACDN-UP3X\\\\u003d\\\",\\\"CAB-AC\\\\u003d\\\"]}]}\",\"timeTaken\":39}",
"@version": "1",
"@timestamp": "2019-06-14T09:25:48.647Z",
"offset": 1354,
"prospector": {
  "type": "log"
 }
}
}

I want to parse the json "host". I want to extract "name" from it into a field called "host" of my own. To break it down into step by step, I first just tried to parse the json by using the following filter

json { source => "host" }

However, when I do this, I get a '_jsonparsefailure' tag in my output in logstash logs

output received {"event"=>{"type"=>"io", "offset"=>2716, "logLevel"=>"INFO ", "input"=>{"type"=>"log"}, "host"=>{"name"=>"lpcservices-dev-int-alln-75f4d6df98-t9wj2"}, "@timestamp"=>2019-06-17T05:45:48.000Z, "serviceType"=>"catalog_ITEM_PUBLISHABLE", "env"=>"STAGE-YS2", "noOfLines"=>0, "timeTaken"=>38, "request"=>"{\"requestHeader\":{\"transactionID\":\"CAE-TEST-15\",\"ccoID\":\"NETFORMX-XML\"},\"entries\":[{\"priceList\":\"WCH2\",\"items\":[\"L-CDACDN-UP3X\\u003d\",\"CAB-AC\\u003d\"]}]}", "timestamp"=>"2019-06-16 22:45:48", "source"=>"/home/jboss/lae-home/app-root/logs/lpcatalogservicesAppIO.log", "tags"=>["applicationLogs", "beats_input_codec_plain_applied", "_jsonparsefailure"], "@version"=>"1", "threadName"=>"IOLogger", "transactionId"=>"CAE-TEST-15", "prospector"=>{"type"=>"log"}}}

Any idea why it's throwing this error?

[Badger]

Can't you just reference [host][name]?

[Arnav_Sengupta]

Okay, that worked. I think I know why my json filter didn't work. The debug log shows the input data as follows

"host"=>{"name"=>"lpcservices-dev-int-alln-75f4d6df98-t9wj2"}

This isn't really a json.

One question, does this mean nested fields from beats can be accessed as associative arrays?

[Badger]

If a field is an object then you can reference fields inside that object, yes. Like [host][name] or [beat][hostname].