Multi line CSV Log

[millerb1]

We have a CSV log and the very last element contains a multi line message. Only the first line appears in Elastic.

input {
beats {
port => 5044
}
stdin {
codec => multiline {
pattern => "\n"
what => "previous"
}
}
}

filter {
csv {
columns => ["local_timestamp","level","code","message"]
separator => "|"
skip_empty_columns => "true"
}
mutate { gsub => ["local_timestamp", "~", ""] }
date {
match => ["local_timestamp", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
}
}

EXAMPLE:
~2018-03-15 13:50:02.876|ERROR|40590DD|Exception #1 Type
System.Data.SqlClient.SqlException

Exception #1 Message
Invalid object name 'Edge_WatchListTypes'.

Exception #1 Stack Trace
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()

Exception #1 Data
HelpLink.ProdName - Microsoft SQL Server
HelpLink.ProdVer - 11.00.6248
HelpLink.EvtSrc - MSSQLServer
HelpLink.EvtID - 208
HelpLink.BaseHelpUrl - http://go.microsoft.com/fwlink
HelpLink.LinkId - 20476

Error Code: -2146232060

[millerb1]

FROM Beats:

2018-03-15T14:44:01.092-0600 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-03-15T20:43:56.088Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.2"
},
"prospector": {
"type": "log"
},
"beat": {
"name": "01SYS52",
"hostname": "01SYS52",
"version": "6.2.2"
},
"source": "C:\ELK\Input\MyLog_20180315.txt",
"offset": 5439,
"message": "~2018-03-15 13:50:02.907|ERROR|613E1C9|Exception #1 Type\nSystem.Data.SqlClient.SqlException\n\nException #1 Message\nInvalid object name 'Edge_WatchListSources'.\n\nException #1 Stack Trace\n at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\n at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\n at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean\u0026 dataReady)\n at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()\n at System.Data.SqlClient.SqlDataReader.get_MetaData()\n at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)\n at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task\u0026 task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)\n at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task\u0026 task, Boolean\u0026 usedCache, Boolean asyncWrite, Boolean inRetry)\n at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)\n at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)\n at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)\n at System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)\n at System.Data.Common.DbDataAdapter.Fill(DataSet dataSet)\n at Edge.DAL.SqlExtensions.ToDataTable(SqlCommand command)\n at Edge.DAL.SqlExtensions.ToDropDownOptions(SqlCommand command)\n at Edge.DAL.WatchListRepo.GetSources()\n at Edge.Controllers.WatchListApiController.GetSources()\n\nException #1 Data\nHelpLink.ProdName - Microsoft SQL Server\nHelpLink.ProdVer - 11.00.6248\nHelpLink.EvtSrc - MSSQLServer\nHelpLink.EvtID - 208\nHelpLink.BaseHelpUrl - http://go.microsoft.com/fwlink\nHelpLink.LinkId - 20476\n",
"tags": [
"component-log"
]
}

[magnusbaeck (Magnus Bäck)]

stdin {
codec => multiline {
pattern => "\n"
what => "previous"
}
}

What's the idea behind this? If the line contains a newline character join it with the previous line? But all lines contain a newline character.

And what's the idea with the multiline codec connected to the stdin input? Are you actually going to use that or are you planning to use the multiline codec with the beats input?