If a single log file has data in different sections and each section is in a different format then how to parse it using logstash

(Nanda Kishore) #1

My log file has different sections. At the beginning it can have a tabular data and then a multiple log lines with format1 and then multiple lines with format 2 etc..
How can I split the log file in to different sections and parse each individual section of lines with a specific way...
For eg.. I want to extract tabular data portion and extract intereseted fields from it using csv filer.
Again in the same log file, I want to extract section having PATTERN1 type lines and parse it using grok filter.
Again in the same log file, I want to extract section having PATTERN2 type lines and parse it using grok filter.

I can identify start of each section in log file by the pattern of the first line.

Is something like this can be achieved using Logstash?


You can have multiple patterns in a single match stanza, example:

match => { "message" => [ "match1 pattern","match2 pattern","match3 pattern" ] }

(Nanda Kishore) #3

Hi Thanks for your reply. I will check that.
But I have a single log file having multiple csv formats and some non csv formats too..Like below
I need parse each section of different kind of csv formats seperately and extract few fields from it. Also I need to extract few fields from non csv lines having different patterns below too..
I know I can use grok with multiple pattern matches for each line..
But in case of csv data, how would i know which line belongs to what csv format?
since I know header rows before hand, is there some thing that I can use header row for each csv format and say that all the content in the same pattern below it belongs it?
Can you please suggest is there good way to parse these kind of files in logstash?

File Start

#Non CSV data..
Motorola System Bios Phoenix NoteBIOS 4.0 Release 6.0

Microsoft Windows Server 2003 Enterprise Edition with Service Pack 2
Microsoft Corporation MSXML 4.0 SP2 Parser and SDK 4.20.9818.0

LSI Logic MegaRC 1.14
LSI Logic MegaRAID SCSI 320-0X RAID Controller Driver

File End

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.