Logstash Filter to convert comma separated syslog message body to JSON format

Elastic Stack Logstash
1

We have one of the application server forwarding syslog to logstash. The "message" body of the log contains few unwanted fields inserted by the application server and remaining required fields as well. Providing below the sample log format.

{
"_index": “applogs",
"_type": "doc",
"_id": "CKsxa2kB00P1Rp_OBCpg",
"_score": 1,
"_source": {
"@timestamp": "2019-03-11T05:19:28.236Z",
"message": "2019-03-11T05:19:28.004Z 10.63.1.223 <54>Mar 11 00:18:38 server06 Action: Virus found,IP Address: 10.1.1.1,Computer name: client01,Intensive Protection Level: 0,Certificate issuer: ,Certificate signer: ,Certificate thumbprint: ,Signing timestamp: 0,Certificate serial number: ,Source: Virus scan,Risk name: EICAR Test String,Occurrences: 1,C:\Users\admin01\Desktop\EICAR-Test01.txt,,Actual action: Quarantined,Requested action: Quarantined,Secondary action: Deleted,Event time: 2019-03-11 00:17:06,Inserted: 2019-03-11 00:18:38,End: 2019-03-11 00:17:06,Last update time: 2019-03-11 00:18:38,Domain: Default,Group: Company\Servers001,Server: EVENTSERVER01,User: admin01,Source computer: ,Source IP: ,Disposition: Good,Download site: ,Web domain: ,Downloaded by: ,Prevalence: This file has been seen by millions of Symantec users.,Confidence: This file is trustworthy.,URL Tracking Status: Off,,First Seen: Reputation was not used in this detection.,Sensitivity: ,Not on the permitted application list,Application hash: 275A021BBFB6489E54D471899F7DB9D1663FC695EC2FE2A2C4538AABF651FD0F,Hash type: SHA2,Company name: ,Application name: eicars.test.com,Application version: ,Application type: 127,File size (bytes): 68,Category set: Malware,Category type: Virus,Location: Default",
"@version": "1"
},
"fields": {
"@timestamp": [
"2019-03-11T05:19:28.236Z"
]
}
}

We need this log to be converted to the below JSON format with only required fields as shown below. Can somebody help with a sample filter which can be used here?

"@timestamp": "2019-03-11T05:19:28.236Z"
"IP Address": " 10.1.1.1"
"Computer name": " client01"
"Intensive Protection Level": " 0"
"Certificate issuer": " "
"Certificate signer": " "
"Certificate thumbprint": " "
"Signing timestamp": " 0"
"Certificate serial number": " "
"Source": " Vius scan"
"Risk name": " EICAR Test String"
"Occurrences": " 1"
"FilePath": " C": "\Users\admin01\Desktop\EICAR-Test0065.txt"
"Actual action": " Quarantined"
"Requested action": " Quarantined"
"Secondary action": " Deleted"
"Event time": " 2019-03-11 00": "17": "06"
"Inserted": " 2019-03-11 00": "18": "38"
"End": " 2019-03-11 00": "17": "06"
"Last update time": " 2019-03-11 00": "18": "38"
"Domain": " Default"
"Group": " Company\Servers001"
"Server": " EVENTSERVER01"
"User": " admin01"
"Source computer": " "
"Source IP": " "
"Disposition": " Good"
"Download site": " "
"Web domain": " "
"Downloaded by": " "
"Prevalence": " This file has been seen by millions of Symantec users."
"Confidence": " This file is trustworthy."
"URL Tracking Status": " Off"
"First Seen": " Reputation was not used in this detection."
"Sensitivity": " "
"Not on the permitted application list"
"Application hash": " 275A021BBFB6489E54D471899F7DB9D1663FC695EC2FE2A2C4538AABF651FD0F"

2

You can parse it using

dissect { mapping => { "message" => "%{[@metadata][ts]} %{} <%{}>%{} %{} %{} %{} %{[@metadata][restOfLine]}" } }
kv { source => "[@metadata][restOfLine]" field_split => "," value_split => ":" }
date { match => [ "[@metadata][ts]", ISO8601 ] }

To limit the fields you could use either include_keys or exclude_keys on the kv filter.

1 Like
3

its giving the "dissectfailure" error and the output is not getting parsed as required.

"@timestamp":"2019-03-11T02:49:31.817Z","tags":["_dissectfailure"]}{"message":"<54>Mar 11 21:47:43 server06 Action: Virus found,IP Address: 10.1.1.1,Computer name: client01,Intensive Protection Level: 0,Certificate issuer: ,...........

4

Your first post said the message format was

"message": "2019-03-11T05:19:28.004Z 10.63.1.223 <54>Mar 11 00:18:38 server06 Action: Virus found,IP Address: 10.1.1.1

The most recent post said it was

<54>Mar 11 21:47:43 server06 Action: Virus found,IP Address: 10.1.1.1

That is a different format and will require a different dissect.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.