Json parse failure even when the json is valid

Elastic Stack Logstash
1

conf filter:
filter{
mutate {
gsub => [
"message","\r","",
"message","\n","",
"message"," ",""
]
}
grok {
match => { "message" => "^%{YEAR:year}-%{MONTH:month}-%{MONTHDAY:day} %{TIME:time} %{GREEDYDATA:pam} [%{GREEDYDATA:data}] %{LOGLEVEL:security} %{GREEDYDATA:class} - %{WORD:jsn} = { %{GREEDYDATA:mth}}" }
}
json {
source => "mth"
target => "parsemethod"
}
if [jsn] != "Message" {
drop { }
}
#if [method] != "getDeviceInfoResponse" {
#drop { }
#}
}

Logstash parser:
{
"@timestamp" => 2018-06-27T11:51:13.948Z,
"@version" => "1",
"mth" => " "getDeviceInfoResponse" : { "messageId" : "123", "message" : "Success", "rudi" : "REF
01234567891234-16", "geoRegion" : "North America", "country" : "US", "status" : "Registered", "connectivit
y" : "To be connected", "sharedAccessKey" : "" }",
"day" => "08",
"month" => "May",
"jsn" => "Message",
"path" => "C:\Users\HOSDURGP\Documents\Varsha\Varsha\rsp-geodir-app.log",
"year" => "2018",
"class" => " com.roche.rsp.geodir.rabbitmq.Publisher",
"tags" => [
[0] "multiline",
[1] "_jsonparsefailure"
],
"data" => "SimpleAsyncTaskExecutor-1",
"security" => "INFO",
"time" => "15:37:24",
"host" => "RDR-VM592",
"pam" => "PM",
"message" => "2018-May-08 15:37:24 PM [SimpleAsyncTaskExecutor-1] INFO com.roche.rsp.geodir.rabbitmq.Publisher -
Message = { "getDeviceInfoResponse" : { "messageId" : "123", "message" : "Success", "rudi" : "REF01234567
891234-16", "geoRegion" : "North America", "country" : "US", "status" : "Registered", "connectivity" : "
To be connected", "sharedAccessKey" : "" }}2018-May-08 15:38:31 PM Routing Key = geodir.msgbroker.cmdprocess"
}

2

The mth field is not valid JSON. It's missing a { at the beginning.

3

Hey Magnus,

Thanks for the reply.
But when I check the mth filed value on json lint, I found that it is a valid json.

"mth" => " "getDeviceInfoResponse" : { "messageId" : "123", "message" : "Success", "rudi" : "REF
01234567891234-16", "geoRegion" : "North America", "country" : "US", "status" : "Registered", "connectivit
y" : "To be connected", "sharedAccessKey" : "" }",

Do you think I should be adding a '{' at the beginning and append it with '}' at the end?

4

FYI

Trying to parse the following lines on the log file:
2018-May-08 15:05:34 PM [SimpleAsyncTaskExecutor-1] INFO com.roche.rsp.geodir.rabbitmq.Subscriber - RECEIVED REQUEST : storeDeviceInfo
2018-May-08 15:05:34 PM [SimpleAsyncTaskExecutor-1] INFO com.roche.rsp.geodir.rabbitmq.Publisher - Message = {
"storeEvent" : {
"eventType" : "DEVICE UPDATE",
"eventName" : "Device registration",
"eventMessage" : "New Device registered, RUDI :REF01234567891234-12",
"timestamp" : "Tue May 08 15:05:34 CEST 2018"
}
}
2018-May-08 15:38:31 PM Routing Key = geodir.msgbroker.eventstore
2018-May-08 15:05:34 PM [SimpleAsyncTaskExecutor-1] INFO com.roche.rsp.geodir.service.GeoDirectoryService - EVENT GENERATED.
2018-May-08 15:05:34 PM [SimpleAsyncTaskExecutor-1] INFO com.roche.rsp.geodir.service.GeoDirectoryService - DEVICE INFO SAVED, RUDI :REF01234567891234-12
2018-May-08 15:05:34 PM [SimpleAsyncTaskExecutor-1] INFO com.roche.rsp.geodir.rabbitmq.Publisher - Message = {
"storeDeviceInfoResponse" : {
"messageId" : "123",
"rudi" : "REF01234567891234-12",
"message" : "SAVED"
}
}

Is it possible for you to help me out with the correct filter to parse the json out of this log file?

Thanks

5

If you format your logs as preformatted text using markdown notation or the </> toolbar button we can see what it actually looks like.

Wait, this part of your grok expression is wrong:

... { %{GREEDYDATA:mth}}

Change it to

... %{GREEDYDATA:mth}

so that the braces are included in the mth field.

6

The reason why I used that grok expression { %{GREEDYDATA:mth}} is because it reads the next two lines along as they don't match the pattern provided in the multiline codec in the input plugin parameters.

Here is how it parses if the grok expression %{GREEDYDATA:mth} is used.
{
"host" => "RDR-VM592",
"jsn" => "Message",
"year" => "2018",
"month" => "May",
"time" => "15:38:21",
"class" => " com.roche.rsp.geodir.rabbitmq.Publisher",
"@timestamp" => 2018-06-27T14:19:50.520Z,
"pam" => "PM",
"mth" => "{ "updateSasKeyResponse" : { "messageId" : "123", "rudi" : "REF01234567891234-16", "mes
sage" : "UPDATED SAS KEY" }}2018-May-08 15:38:31 PM Routing Key = geodir.msgbroker.cmdprocess2018-May-08 15:38:31 PM
[SimpleAsyncTaskExecutor-1] INFO com.roche.rsp.geodir.rabbitmq.Subscriber - RECEIVED REQUEST : updateSasKey2018-May-08
15:38:31 PM [SimpleAsyncTaskExecutor-1] ERROR com.roche.rsp.geodir.service.GeoDirectoryService - FAILED TO UPDATE SAS K
EY, RUDI :REF01234567891234-16",
"message" => "2018-May-08 15:38:21 PM [SimpleAsyncTaskExecutor-1] INFO com.roche.rsp.geodir.rabbitmq.Publisher -
Message = { "updateSasKeyResponse" : { "messageId" : "123", "rudi" : "REF01234567891234-16", "message" :
"UPDATED SAS KEY" }}2018-May-08 15:38:31 PM Routing Key = geodir.msgbroker.cmdprocess2018-May-08 15:38:31 PM [SimpleAs
yncTaskExecutor-1] INFO com.roche.rsp.geodir.rabbitmq.Subscriber - RECEIVED REQUEST : updateSasKey2018-May-08 15:38:31
PM [SimpleAsyncTaskExecutor-1] ERROR com.roche.rsp.geodir.service.GeoDirectoryService - FAILED TO UPDATE SAS KEY, RUDI :
REF01234567891234-16",
"@version" => "1",
"tags" => [
[0] "multiline",
[1] "_jsonparsefailure"
],
"data" => "SimpleAsyncTaskExecutor-1",
"path" => "C:\Users\xxxx\xxxx\xxxxx\xxxxxx\rsp-geodir-app.log",
"security" => "INFO",
"day" => "08"
}

7

Why is your multiline config joining multiline lines into the same Logstash event? That seems broken.

I repeat: If you format your logs as preformatted text using markdown notation or the </> toolbar button we can see what they actually look like.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.