Fixed The Json But Logstash still produces Jsonparseerror

Hello There
I have logs coming in to syslogng. But some of the logs do not have the correct Json structure.

This is the error log:

[2018-10-16T07:40:32,778][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at 

This is the log incoming:
<01>- hostname{
"name":"Sysmon",
"version":"1.0",
"isoTimeFormat":"yyyy-MM-dd'T'HH:mm:ss.SSSZ",
"type":"Event",
"ParentImageName":"ccSvcHst.exe",
"ImageName":"RegSSHelper.exe",
"Process CommandLine":""C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\14.0.3929.1200.105\Bin64\RegSSHelper.exe" "C:\ProgramData\Symantec\Symantec Endpoint Protection\14.0.3929.1200.105\Data\Definitions\HIDefs\20171204.001\SSHelper64.dll"",
"SHA256 Hash":"F4DB5D19A6D1EF5F55995F57E1C79E7EEE4AC4066" [
truncated 726 chars
] ; line:1,
column:2
]>,
:data=>"<01>- hostname{
"name":"Sysmon",
"version":"1.0",
"isoTimeFormat": "yyyy-MM-dd'T'HH: mm:ss.SSSZ",
"type":"Event",
"ParentImageName":"ccSvcHst.exe",
"ImageName":"RegSSHelper.exe",
"Process CommandLine": "\"C: \\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin64\\RegSSHelper.exe\" \"C:\\ProgramData\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Data\\Definitions\\HIDefs\\20171204.001\\SSHelper64.dll\"",
"SHA256 Hash":"F4DB5D19A6D1EF5F55995F57E1C79E7EEE4AC406637EFC3A41809A111B4B0563",
"File Hash":"A0100CB0B991EE012A712831638F4579",
"ParentImage": "C:\\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin\\ccSvcHst.exe",
"ParentCommandLine": "\"C: \\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin\\ccSvcHst.exe\" /s \"Symantec Endpoint Protection\" /m \"C: \\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin\\sms.dll\" /prefetch:1 ",
"Image": "C:\\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin64\\RegSSHelper.exe",
"Parent Process Guid":"964EB05A-7CC1-5BC5-0000-0010A3570400",
"Process Guid":"964EB05A-7CD7-5BC5-0000-001076271400",
"Process Id":"8180"
}

As you can see there is "<01>- hostname " that makes JSON unstructured (The log source sends logs this way).

However I remove them with the mutate:

input {
  tcp {
    port => 6050
    codec =>   json
    tags => "syslogng"
  }
}
filter {
  if "syslogng" in [tags] {
        mutate {
          #add_tag => [ "conf_file_0000"]
        }
  }
  if "hostname" in [message] {
        mutate {
          gsub => ["message", "<01>- hostname ", ""]
        }
  }
}

Removing it is successfull since I can see that"

{  
   \"name\":\"Sysmon\",
   \"version\":\"1.0\",
   \"isoTimeFormat\":   \"yyyy-MM-dd'T'HH:   mm:ss.SSSZ\",
   \"type\":\"Event\",
   \"ParentImageName\":\"svchost.exe\",
   \"ImageName\":\"dllhost.exe\",
   \"Process CommandLine\":   \"C:   \\\\Windows\\\\SysWOW64\\\\DllHost.exe /Processid:{  
      E2B3C97F-6AE1-41AC-817A-F6F92166D7DD
   }   \",
   \"SHA256 Hash\":\"F7AD4B09AFB301CE46DF695B22114331A57D52E6D4163FF74787BF68CCF44C78\",
   \"File Hash\":\"A63DC5C2EA944E6657203E0C8EDEAF61\",
   \"ParentImage\":   \"C:\\\\Windows\\\\System32\\\\svchost.exe\",
   \"ParentCommandLine\":   \"C:\\\\Windows\\\\system32\\\\svchost.exe -k DcomLaunch\",
   \"Image\":   \"C:\\\\Windows\\\\SysWOW64\\\\dllhost.exe\",
   \"Parent Process Guid\":\"6952E6F6-D150-5BA8-0000-0010F4FD0000\",
   \"Process Guid\":\"6952E6F6-951A-5BC5-0000-0010F36C746E\",
   \"Process Id\":\"4872\"
}

However this error is still produced on logstash logs.
What can I do to preprocess the logs before the JSON is tagged with jsonparsefailiure tag so that parsed values can be indexed correctly.?

Regards.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.