Multiline log files

Elastic Stack Beats
1

I am looking to configure filebeats to deal with the following data sample...

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Time of Server Script Exception : 26 January 2018 13:38:02

1: Source:.Net SqlClient Data Provider
1: Type :System.Data.SqlClient.SqlException
1: Message:Incorrect syntax near the keyword 'VIEW'.
Incorrect syntax near the keyword 'AS'.
1: Target Site:System.String LoadXML(Int32, Int64, Int64, Int64, System.String, System.String, Int32, System.String, System.String, System.String, System.String, System.String, Int32)
1: Stack Trace: at XXX.YYY.clsBizSummaryPanel.LoadXML(Int32 intSummaryPanelID, Int64 lngParentID, Int64 lngPatientID, Int64 lngUserID, String strLanguageCode, String strParam1, Int32 intDataSourceID, String strtempPatList, String strDataID, String streRxAccountStatusCount, String strSectionXml, String strPreferredSettings, Int32 IntpgqaeSize)
1: at XXX.ZZZ.ZZZ.LoadXML(Int32 intSummaryPanelID, Int64 lngParentID, Int64 lngPatientID, Int64 lngUserID, String strLanguageCode, String strParam1, Int32 intDataSourceID, String strTempPatlist, String strDataID, String streRxAccountStatusCount, String strSectionXml, String strPreferredSettings, Int32 intPageSize)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Time of Server Script Exception : 26 January 2018 13:42:02

1: Source:.Net SqlClient Data Provider
1: Type :System.Data.SqlClient.SqlException
1: Message:Incorrect syntax near the keyword 'VIEW'
1: Target Site:System.String LoadXML(Int32, Int64, Int64, Int64, System.String, System.String, Int32, System.String, System.String, System.String, System.String, System.String, Int32)
1: Stack Trace: at XXX.YYY.clsBizSummaryPanel.LoadXML(Int32 intSummaryPanelID, Int64 lngParentID, Int64 lngPatientID, Int64 lngUserID, String strLanguageCode, String strParam1, Int32 intDataSourceID, String strtempPatList, String strDataID, String streRxAccountStatusCount, String strSectionXml, String strPreferredSettings, Int32 IntpgqaeSize)
1: at XXX.ZZZ.ZZZ.LoadXML(Int32 intSummaryPanelID, Int64 lngParentID, Int64 lngPatientID, Int64 lngUserID, String strLanguageCode, String strParam1, Int32 intDataSourceID, String strTempPatlist, String strDataID, String streRxAccountStatusCount, String strSectionXml, String strPreferredSettings, Int32 intPageSize)

So, the data we wish to harvest from the first data set is...

Field Name:Time of Server Script Exception
Data: 26 January 2018 13:38:02

Field Name:Source
Data:.Net SqlClient Data Provider

Field Name:Type
Data:System.Data.SqlClient.SqlException

Field Name:Message
Data:Incorrect syntax near the keyword 'VIEW'.Incorrect syntax near the keyword 'AS'.

Field Name:Target Site
Data:System.String LoadXML

Field Name:Stack Trace
Data:at XXX.YYY.clsBizSummaryPanel.LoadXML at XXX.ZZZ.ZZZ.LoadXML

Can someone help.

Thanks

2

Have you looked at the documentation around this? https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html

3

Hi, I have had a look at this.
In addition to the multiline, I have a few other queries...

A) How would I ignore the first line, ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

B) How can I pick out the date from the string in the second line, Time of Server Script Exception : 26 January 2018 13:38:02

C) How can I configure to show each new entry starts with the line, ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

4

UPDATE
Using multiline syntax and exclude line...
I could exclude all lines starting with '++++++' then take all the lines that do not start with Time of Server Script Exception : and combine them with the previous line that does.

This would leave me with an event that starts …

'Time of Server Script Exception : 26 January 2018 13:38:02 1: Source:.Net SqlClient Data Provider 1: Type :System.Data.SqlClient.SqlException 1: Message:Incorrect syntax near the keyword 'VIEW'. Incorrect syntax near the keyword 'AS'. 1: Target Site:System.String LoadXML(Int32, Int64, Int64, Int64, System.String, System.String, Int32, System.String, System.String, System.String, System.String, System.String, Int32)……...'

I could then try and work out a pattern to extract the correct data items, e.g. '26 January 2018 13:38:02' and '.Net SqlClient Data Provider' etc.

How does this approach seem?

5

Sounds like you're on the right track!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.