Logstash warning

Elastic Stack Logstash
1

i get this warning in logstash log

[2017-11-13T10:41:22,803][WARN ][logstash.filters.json ] Error parsing json {:source=>"LogMsg", :raw=>"UID:420: Out /getData/{city}/{speciality}/{doctorName} all/all/all", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'UID': was expecting ('true', 'false' or 'null') at [Source: [B@7139abf; line: 1, column: 5]>}

2

You've asked Logstash to parse

UID:420: Out /getData/{city}/{speciality}/{doctorName} all/all/all

as JSON but it's not a JSON string.

1 Like
3

i have the valid json log lines.Still getting this warning

4

The LogMsg field that you're trying to parse did in this case not contain a valid JSON string. Period.

Perhaps you already have a json or json_lines codec in your input configuration, making the json filter unnecessary.

1 Like
5

Ohh,Then shall i remove the stdin { codec => "json" } from my input filter?

6

Either that, or remove the json filter. Just don't keep both.

1 Like
7

i removed the stdin { codec => "json" } in input filter but getting the warning in logstash

8

my sample log format
{"LogLevel":"ERROR","LogMsg":"{\"itemId\":0,\"module\":\"/curie/encounter\",\"action\":\"/addToken\",\"errorMessage\":\"java.lang.RuntimeException: org.apache.ibatis.exceptions.PersistenceException: \\n### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.\\n### The error may exist in EncounterMapper.xml\\n### The error may involve EncounterMapper.checkTokenExist-Inline\\n### The error occurred while setting parameters\\n### SQL: SELECT COUNT(*) FROM token WHERE appointmentId \\u003d ?\\n### Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.\",\"parameter\":\"java.lang.RuntimeException: java.lang.RuntimeException: org.apache.ibatis.exceptions.PersistenceException: \\n### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.\\n### The error may exist in EncounterMapper.xml\\n### The error may involve EncounterMapper.checkTokenExist-Inline\\n### The error occurred while setting parameters\\n### SQL: SELECT COUNT(*) FROM token WHERE appointmentId \\u003d ?\\n### Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.systems.ellora.core.api.encounter.domain.EncounterBuilder.addToken(EncounterBuilder.java:995)\"}","Time":"2017-11-20_09:12:21.042"} {"LogLevel":"INFO","LogMsg":"UID: Invoice [invoiceId=465, encounterId=1676, subject=null, reason=null, dateRaised=null, total=299, isPaid=false, datePaid=null, lineItems=[LineItem [invoiceId=0, consultancy=null, labFee=null, scanFee=null, vaccination=null, medicalDispense=MedDispense [invoiceId=465, medicationId=104, totalAmount=49, performer=v7LoV3ZFhLZy0z8M4uHpAJisTeJ3, type=Normal, facilityId=3, doctorId=3, encounterId=1676, patientKey=17169, listOfItem=[DispenseItem [medicationId=104, substanceDetailId=56, substanceName=Ambroxol hydrochloride, quantity=1, cost=49.0]]], chargeItem=null], LineItem [invoiceId=0, consultancy=ConsultancyFee [name=Consultancy, amount=250, performer=null], labFee=null, scanFee=null, vaccination=null, medicalDispense=null, chargeItem=null], LineItem [invoiceId=0, consultancy=null, labFee=null, scanFee=ScanFee [name=ScanFee, amount=1000, performer=null], vaccination=null, medicalDispense=null, chargeItem=null], LineItem [invoiceId=0, consultancy=null, labFee=LabFee [name=LabFee, amount=500, performer=null], scanFee=null, vaccination=null, medicalDispense=null, chargeItem=null]]], Completed Invoice Task {}UID:11: ","Time":"2017-11-20_12:11:07.931"}

im trying to remove the "LogLevel"="INFO"
and i want only "LogLevel":"ERROR" field to shown in kibana

My filter plugin :
filter { json { source => "message" } json { source => "LogMsg" } if [LogLevel] == "INFO" { drop { remove_field => [ "LogLevel" ] } } mutate { add_field => { "ErrorMsg" => "%{errorMessage}" } } truncate { fields => "ErrorMsg" length_bytes => 1000 } }

Correct me if I'm wrong

9

Your first sample line does indeed have a LogMsg field that's JSON, but the second one doesn't. Perhaps you should use the json filter's skip_on_invalid_json option.

2 Likes
10

Thank you so much, appreciate your help

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.