HI,
I am loading data from a text file to logstash using multiline grok, but every time stamp is creating for different fields, can you please help me here
Input file:
DATE: 2019-12-03
Input File/Folder:1.txt
SHA256 hash of 1.txt: 2f1d3f3343dbc60768a290565c9136f754015438e4ea25770b61e624e6443792
Initiator Name: req1@hpe.com
OS:Linux
Scanning Tools: McAfee & ClamAV
*********************!!! MCAFEE REPORT ON LINUX!!! **********************
McAfee VirusScan Command Line for Linux64 Version: 6.0.6.653
Copyright (C) 2015 McAfee, Inc.
(408) 988-3832 LICENSED COPY - March 14 2016
AV Engine version: 5800.7501 for Linux64.
Dat set version: 9458 created Dec 1 2019
Scanning for 668685 viruses, trojans and variants.
2019-Dec-03 23:21:47
Options:
--AFC= 512 --ANALYZE --IGNORE-LINKS --MIME --SUB --SUMMARY --VERBOSE --ATIME-PRESERVE --REPORT=/AVScan_Logs/Linux/McAfee_1.txt.log --UNZIP --THREADS=4 --CLEAN /data/mysftpuser/MalwareScan/1.txt
/data/mysftpuser/MalwareScan/1.txt ... is OK.
Summary Report on /data/mysftpuser/MalwareScan/1.txt
File(s)
Total files:................... 111
Clean:......................... 222
Not Scanned:................... 333
Possibly Infected:............. 444
Cleaned:....................... 555
Deleted:....................... 666
Time: 00:00.00
*********************!!! ClamAV(OpenSource) REPORT ON LINUX!!! **********************
-------------------------------------------------------------------------------
/data/mysftpuser/MalwareScan/1.txt: Empty file
----------- SCAN SUMMARY -----------
Known viruses: 6578161
Engine version: 0.101.4
Scanned directories: 777
Scanned files: 888
Infected files: 999
Data scanned: 0123.00 MB
Data read: 345.00 MB (ratio 0.00:1)
Time: 33.534 sec (0 m 33 s)
====================
we need seperate fields for most of the lines to use it in Kibana graphs, can some help me here pls
My Config file:
input {
file {
path => "/root/elk/data/mslog"
start_position => "beginning"
codec => multiline {
#pattern => "(^\s*)"
pattern => "((^\s*)|(\n))"
negate => true
what => "previous"
}
}
}
filter {
if [message] =~ "DATE:" {
grok {
match => { "message" => "(20%{DATE:startdate})" }
}
} else if [message] =~ "Input File/Folder:" {
grok {
match => { "message" => "%{GREEDYDATA:filefolder}" }
}
} else if [message] =~ "SHA256 hash of" {
grok {
match => { "message" => ":%{GREEDYDATA:shahid}" }
}
} else if [message] =~ "Initiator Name:" {
grok {
==================
totalfiles scanneddirectories scannedfiles scanningtools scantime shahid startdate options notscanedfiles knownviruses infectedfiles filefolder engineversion deleted
Dec 23, 2019 @ 15:20:15.638 - - - - - - - - - - - - - -
Dec 23, 2019 @ 15:20:15.638 - - - - - - - - - - - - - -
Dec 23, 2019 @ 15:20:15.637 - - - - - - - - - - 999 - - -
Dec 23, 2019 @ 15:20:15.637 - - 888 - - - - - - - - - - -
Dec 23, 2019 @ 15:20:15.636 - - - - - - - - - 6578161 - - - -
Dec 23, 2019 @ 15:20:15.636 - - - - - - - - - - - - 0.101 -
Dec 23, 2019 @ 15:20:15.636 - 777 - - - - - - - - - - - -
Dec 23, 2019 @ 15:20:15.635 - -
{
"@timestamp" => 2019-12-23T10:03:16.376Z,
"@version" => "1",
"path" => "/root/elk/data/mslog",
"message" => "\tData scanned: 0123.00 MB",
"host" => "securedev31.in.rdlabs.hpecorp.net"
}
{
"@version" => "1",
"host" => "securedev31.in.rdlabs.hpecorp.net",
"@timestamp" => 2019-12-23T10:03:16.346Z,
"path" => "/root/elk/data/mslog",
"message" => "\tData read: 0.00 MB (ratio 0.00:1)",
"dataread" => "\tData read: 0.00 MB (ratio 0.00:1)"
}
{
"@version" => "1",
"host" => "securedev31.in.rdlabs.hpecorp.net",
"@timestamp" => 2019-12-23T10:03:16.376Z,
"path" => "/root/elk/data/mslog",
"message" => "\tData read: 345.00 MB (ratio 0.00:1)",
"dataread" => "\tData read: 345.00 MB (ratio 0.00:1)"
}
{
"@version" => "1",
"host" => "securedev31.in.rdlabs.hpecorp.net",
"@timestamp" => 2019-12-23T10:03:16.348Z,
"path" => "/root/elk/data/mslog",
"message" => "\tTime: 33.534 sec (0 m 33 s)",
"scantime" => "\tTime: 33.534 sec (0 m 33 s)"
}
{
"@version" => "1",
"host" => "securedev31.in.rdlabs.hpecorp.net",
"@timestamp" => 2019-12-23T10:03:16.370Z,
"path" => "/root/elk/data/mslog",
"message" => "\tTime: 00:00.00",
"scantime" => "\tTime: 00:00.00"
}