JSON Parsing failure on logstash

Elastic Stack Logstash
1

HI,
I have json when i try to index those logs into elasticsearch. When i apply json filter some logs are omitted to get index on ES. when i command those json filter i can index those event into ES. i have validated that json input into "https://jsonlint.com/" and it shows valid json. Please find my input and error message which i got,

If i enable the json filter im getting this error , if i disable i'm not getting this error

Input:

{"version":"1.0.0","environment":{"name":"E1","hostName":"txnmessaging-deployment-11-g1d77","virtualMachine":"na","clusterName":"txnmessaging","containerId":"na","containerName":"na","containerType":"OPENSHIFT-JAVA"},"application":{"project":"rtpe","portfolio":"payment-network","name":"txnmessaging"},"type":"EVENT","level":"INFO","message":"Txn Log Messaging is up and running!","fileName":"com.amex.txnlog.handler.HealthCheckHandler","methodName":"checkHealth","lineNumber":25,"thread":"38-nmsworkerverticle-1","timestamp":"2018-11-22T02:40:56.012Z","correlationId":"txnmessaging-deployment-11-g1d77-096980c0-ee00-11e8-b0e4-a7ed1c84c14e","additionalFields":{"X-AXP-UUID":"txnmessaging-deployment-11-g1d77-096980c0-ee00-11e8-b0e4-a7ed1c84c14e","APP_VERSION":"missing app version"}}

Error:

[2018-11-26T10:38:01,725][WARN ][logstash.filters.json    ] Error parsing json {:source=>"lomo_msg", :raw=>"{\"app_name\": \"client-state-mgr\", \"pod_ip\": \"192.168.33.25\", \"time\": \"2018-11-26T10:37:53.885807547Z\", \"environment\": \"e1\", \"pod_name\": \"client-state-mgr-deployment-23-lr3n2\", \"container_name\": \"k8s_client-state-mgr_client-state-mgr-deployment-23-lr3n2_nemo-demo_abdfdb1a-e9d0-11e8-813d-fa163e7ac2b7_14\", \"message\": \"\tat akka.persistence.AbstractPersistentActor.aroundReceive(PersistentActor.scala:454) ~[client-state-management-local-dev-SNAPSHOT-all.jar:?]\", \"ns\": \"nemo-demo\", \"pod_container\": \"client-state-mgr\"}", :exception=>#<LogStash::Json::ParserError: Illegal unquoted character ((CTRL-CHAR, code 9)): has to be escaped using backslash to be included in string value
 at [Source: (byte[])"{"app_name": "client-state-mgr", "pod_ip": "192.168.33.25", "time": "2018-11-26T10:37:53.885807547Z", "environment": "e1", "pod_name": "client-state-mgr-deployment-23-lr3n2", "container_name": "k8s_client-state-mgr_client-state-mgr-deployment-23-lr3n2_nemo-demo_abdfdb1a-e9d0-11e8-813d-fa163e7ac2b7_14", "message": "        at akka.persistence.AbstractPersistentActor.aroundReceive(PersistentActor.scala:454) ~[client-state-management-local-dev-SNAPSHOT-all.jar:?]", "ns": "nemo-demo", "pod_container": "cli"[truncated 15 bytes]; line: 1, column: 318]>}
2

It seems the issue lies with the "message" field, are you sure there is not another log that has a quote inside the message field?

3

if i replace the "message" keyword into "msg" by using gsub it will work or not?

4

I think it is saying that within the message field that there is an extra quote mark that has not been escaped, therefor it causes it to think there is an extra quotation. Are you sure that the input you posted above is the one causing an error?

5

I cross checked on my message and there is no extra quotes or new line.

yes i'm sure when i enable the json filter i'm getting this error on log.

6

The error and the message you are providing me do not match up, for example check the error log:

"container_name": "k8s_client-state-mgr_client-state-mgr-deployment-23-lr3n2_nemo-demo_abdfdb1a-e9d0-11e8-813d-fa163e7ac2b7_14"

Yet you are saying that input is:

"containerName":"na"

Notice the difference? So please provide the correct input

7

{"app_name": "client-state-mgr", "pod_ip": "192.168.33.25", "time": "2018-11-26T12:29:35.608142382Z", "environment": "e1", "pod_name": "client-state-mgr-deployment-23-lr3n2", "container_name": "k8s_client-state-mgr_client-state-mgr-deployment-23-lr3n2_nemo-demo_abdfdb1a-e9d0-11e8-813d-fa163e7ac2b7_14", "message": " at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:222) ~[client-state-management-local-dev-SNAPSHOT-all.jar:?]", "ns": "nemo-demo", "pod_container": "client-state-mgr"}

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.