Hi
I am trying to parse a multi-line log file(CSV) using Logstash/grok pattern. The pattern works inside grok debugger but when i run it in the logstash, it produces _grokparsefailuer. The _grokparsefailuer is there when i remove the CSV header as well.
The log line is bit long, I am pasting only a single line here with the CSV header.
Logfile
---------
PartitionKey,RowKey,Timestamp,Component,Component@type,Created,Created@type,CustomerName,CustomerName@type,ErrorMessage,ErrorMessage@type
1,09cca1e3-5e85-4d58-adca-11f8820ba70e,2018-02-09T16:42:08.775Z,TT.Core.Api,Edm.String,2018-02-08T11:18:22.631Z,Edm.DateTime,BTPL,Edm.String,"System.InvalidOperationException: Source sequence contains more than one element.
at System.Linq.AsyncEnumerable.<SingleOrDefault_>d__381`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.TaskResultAsyncEnumerable`1.Enumerator.<MoveNext>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.<MoveNext>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextExceptionFilterAsync>d__23.MoveNext()",Edm.String
The logstash config file
input {
file {
path => "/home/data/apierror.csv"
start_position => "beginning"
sincedb_path => "/tmp/sincedb"
}
}
filter {
grok {
match=>["message","%{INT:PartitionKey}\,%{DATA:RowKey}\,%{TIMESTAMP_ISO8601:Timestamp}\,%{DATA:Component}\,%{DATA:ComponentType}\,%{TIMESTAMP_ISO8601:Created}\,%{DATA:CreatedType}\,%{DATA:CustomerName}\,%{DATA:CustomerNameType}\,%{QUOTEDSTRING:ErrorMessage}\,%{GREEDYDATA:ErrorMessageType}"]
}
}
output{
stdout { codec => "rubydebug" }
}
I can see from the debug line grok compiled is OK.
[2018-02-19T04:38:06,550][DEBUG][logstash.filters.grok ] Grok compiled OK {:pattern=>"%{INT:PartitionKey}\\,%{DATA:RowKey}\\,%{TIMESTAMP_ISO8601:Timestamp}\\,%{DATA:Component}\\,%{DATA:ComponentType}\\,%{TIMESTAMP_ISO8601:Created}\\,%{DATA:CreatedType}\\,%{DATA:CustomerName}\\,%{DATA:CustomerNameType}\\,%{QUOTEDSTRING:ErrorMessage}\\,%{GREEDYDATA:ErrorMessageType}", :expanded_pattern=>"(?<INT:PartitionKey>(?:[+-]?(?:[0-9]+)))\\,(?<DATA:RowKey>.*?)\\,(?<TIMESTAMP_ISO8601:Timestamp>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)\\,(?<DATA:Component>.*?)\\,(?<DATA:ComponentType>.*?)\\,(?<TIMESTAMP_ISO8601:Created>(?:(?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)\\,(?<DATA:CreatedType>.*?)\\,(?<DATA:CustomerName>.*?)\\,(?<DATA:CustomerNameType>.*?)\\,(?<QUOTEDSTRING:ErrorMessage>(?>(?<!\\\\)(?>\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>
(?>\\.|[^\\]+)+
)|``)))\,(?GREEDYDATA:ErrorMessageType.*)"}`
and thereafter in the logfile, i can see the following lines for the last field(ErrorMessage@type) which is a multiline.
[2018-02-19T04:38:07,591][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/data/apierror.csv", :text=>"1,09cca1e3-5e85-4d58-adca-11f8820ba70e,2018-02-09T16:42:08.775Z,TT.Core.Api,Edm.String,2018-02-08T11:18:22.631Z,Edm.DateTime,BTPL,Edm.String,\"System.InvalidOperationException: Source sequence contains more than one element.\r"}
[2018-02-19T04:38:07,950][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-02-19T04:38:08,005][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/data/apierror.csv", :text=>" at System.Linq.AsyncEnumerable.<SingleOrDefault_>d__381`1.MoveNext()\r"}
[2018-02-19T04:38:08,015][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/data/apierror.csv", :text=>"--- End of stack trace from previous location where exception was thrown ---\r"}
[2018-02-19T04:38:08,021][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/data/apierror.csv", :text=>" at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r"}
[2018-02-19T04:38:08,027][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/data/apierror.csv", :text=>" at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r"}
[2018-02-19T04:38:08,035][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/data/apierror.csv", :text=>" at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r"}
[2018-02-19T04:38:08,041][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/data/apierror.csv", :text=>" at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\r"}
[2018-02-19T04:38:08,051][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/data/apierror.csv", :text=>" at Microsoft.EntityFrameworkCore.Query.Internal.AsyncLinqOperatorProvider.TaskResultAsyncEnumerable`1.Enumerator.<MoveNext>d__3.MoveNext()\r"}
[2018-02-19T04:38:08,061][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/data/apierror.csv", :text=>"--- End of stack trace from previous location where exception was thrown ---\r"}
I am attaching the logstash (debug) log file here.
Can anyone plz. point me where am i doing wrong ?
Regards