How to send log that has many line breaks as one entry


#1

Please let me question how to collect Windows Defender log using logstash.

Windows Defender output log such as the below sample when virus is detected.
But the log has a lot of a line breaks in one detection log.
So logstash send a lot of log to elasticsearch.
Please tell me how to send the below log as one entry.

@sample

Begin Quick Scan
Scan ID:{ECAB1DAA-924B-468D-AEB9-6FCEEBD153D1}
Scan Source:2
Start Time:日 10 11 2015 14:04:42
End Time:日 10 11 2015 14:07:13
Result Count:1
Threat Name:Tool:Win32/EICAR_Test_File
ID:17463
Severity:5
Number of Resources:3
Resource Schema:file
Resource Path:C:\Users\Administrator\Desktop\X5O!P%_AP[4_PZX54_P^7CC_7$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H.txt
Extended Info:5866324352432
Resource Schema:file
Resource Path:C:\Users\Administrator\Desktop\3.txt
Extended Info:5866324352432
Resource Schema:file
Resource Path:C:\Users\Administrator\Desktop\2.txt
Extended Info:5866324352432
End Scan



(Magnus Bäck) #2

Use a multline filter or multiline codec to join consecutive lines. The multiline logic would be something like "unless the line begins with 'Begin Quick Scan', join the current line with the previous line".


#3

Hi Magnusbaeck,

Thank you for your reply !!

I set logstash.conf as the below.

**** @ logstash.conf ****
input {
file {
path => "C:/ProgramData/Microsoft/Windows Defender/Support/MPLog-*.log"
tags => "AntiVirus"
type => 'WindowsDefender'
}

}

filter {
multiline {
type => 'WindowsDefender'
pattern => "Begin Quick Scan"
what => "previous"
}
}

output {
stdout {}
}
**** @ logstash.conf ****

But it seems to not be filtered.
How should I modify logstash.conf ?

BTW, Windows Defender Log start from a line break as the below.


Begin Quick Scan
Scan ID:{ECAB1DAA-924B-468D-AEB9-6FCEEBD153D1}
Scan Source:2
Start Time:日 10 11 2015 14:04:42
End Time:日 10 11 2015 14:07:13
Result Count:1
Threat Name:Tool:Win32/EICAR_Test_File
ID:17463
Severity:5
Number of Resources:3
Resource Schema:file
Resource Path:C:\Users\Administrator\Desktop\X5O!P%_AP[4_PZX54_P^7CC_7$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H.txt
Extended Info:5866324352432
Resource Schema:file
Resource Path:C:\Users\Administrator\Desktop\3.txt
Extended Info:5866324352432
Resource Schema:file
Resource Path:C:\Users\Administrator\Desktop\2.txt
Extended Info:5866324352432
End Scan


Best regards,
Tyler


(Magnus Bäck) #4

You're missing the negation in "unless the line begins with 'Begin Quick Scan', join the current line with the previous line". Add negate => true to the multiline filter.

Also, don't use type in the filter. That form has been deprecated for a long time and will be removed completely in Logstash 2.0. Wrap the filter in a conditional instead.

if [type] == "WindowsDefender" {
  multiline {
     ...
  }
}

#5

Thank you for your description.
I can filter by your procedure.


(system) #6