Fixed The Json But Logstash still produces Jsonparseerror

Elastic Stack Logstash
1

Hello There
I have logs coming in to syslogng. But some of the logs do not have the correct Json structure.

This is the error log:

[2018-10-16T07:40:32,778][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at

This is the log incoming:
<01>- hostname{
"name":"Sysmon",
"version":"1.0",
"isoTimeFormat":"yyyy-MM-dd'T'HH:mm:ss.SSSZ",
"type":"Event",
"ParentImageName":"ccSvcHst.exe",
"ImageName":"RegSSHelper.exe",
"Process CommandLine":""C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\14.0.3929.1200.105\Bin64\RegSSHelper.exe" "C:\ProgramData\Symantec\Symantec Endpoint Protection\14.0.3929.1200.105\Data\Definitions\HIDefs\20171204.001\SSHelper64.dll"",
"SHA256 Hash":"F4DB5D19A6D1EF5F55995F57E1C79E7EEE4AC4066" [
truncated 726 chars
] ; line:1,
column:2
]>,
:data=>"<01>- hostname{
"name":"Sysmon",
"version":"1.0",
"isoTimeFormat": "yyyy-MM-dd'T'HH: mm:ss.SSSZ",
"type":"Event",
"ParentImageName":"ccSvcHst.exe",
"ImageName":"RegSSHelper.exe",
"Process CommandLine": "\"C: \\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin64\\RegSSHelper.exe\" \"C:\\ProgramData\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Data\\Definitions\\HIDefs\\20171204.001\\SSHelper64.dll\"",
"SHA256 Hash":"F4DB5D19A6D1EF5F55995F57E1C79E7EEE4AC406637EFC3A41809A111B4B0563",
"File Hash":"A0100CB0B991EE012A712831638F4579",
"ParentImage": "C:\\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin\\ccSvcHst.exe",
"ParentCommandLine": "\"C: \\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin\\ccSvcHst.exe\" /s \"Symantec Endpoint Protection\" /m \"C: \\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin\\sms.dll\" /prefetch:1 ",
"Image": "C:\\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\14.0.3929.1200.105\\Bin64\\RegSSHelper.exe",
"Parent Process Guid":"964EB05A-7CC1-5BC5-0000-0010A3570400",
"Process Guid":"964EB05A-7CD7-5BC5-0000-001076271400",
"Process Id":"8180"
}

As you can see there is "<01>- hostname " that makes JSON unstructured (The log source sends logs this way).

However I remove them with the mutate:

input {
  tcp {
    port => 6050
    codec =>   json
    tags => "syslogng"
  }
}
filter {
  if "syslogng" in [tags] {
        mutate {
          #add_tag => [ "conf_file_0000"]
        }
  }
  if "hostname" in [message] {
        mutate {
          gsub => ["message", "<01>- hostname ", ""]
        }
  }
}

Removing it is successfull since I can see that"

{  
   \"name\":\"Sysmon\",
   \"version\":\"1.0\",
   \"isoTimeFormat\":   \"yyyy-MM-dd'T'HH:   mm:ss.SSSZ\",
   \"type\":\"Event\",
   \"ParentImageName\":\"svchost.exe\",
   \"ImageName\":\"dllhost.exe\",
   \"Process CommandLine\":   \"C:   \\\\Windows\\\\SysWOW64\\\\DllHost.exe /Processid:{  
      E2B3C97F-6AE1-41AC-817A-F6F92166D7DD
   }   \",
   \"SHA256 Hash\":\"F7AD4B09AFB301CE46DF695B22114331A57D52E6D4163FF74787BF68CCF44C78\",
   \"File Hash\":\"A63DC5C2EA944E6657203E0C8EDEAF61\",
   \"ParentImage\":   \"C:\\\\Windows\\\\System32\\\\svchost.exe\",
   \"ParentCommandLine\":   \"C:\\\\Windows\\\\system32\\\\svchost.exe -k DcomLaunch\",
   \"Image\":   \"C:\\\\Windows\\\\SysWOW64\\\\dllhost.exe\",
   \"Parent Process Guid\":\"6952E6F6-D150-5BA8-0000-0010F4FD0000\",
   \"Process Guid\":\"6952E6F6-951A-5BC5-0000-0010F36C746E\",
   \"Process Id\":\"4872\"
}

However this error is still produced on logstash logs.
What can I do to preprocess the logs before the JSON is tagged with jsonparsefailiure tag so that parsed values can be indexed correctly.?

Regards.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.