Can someone please tell me how I can debug this pattern and corresponding log entry that works on the grok debugger but fails in logstash
Here is the code that works on the grok debugger:
(.*)\]%{SPACE}[-]%{SPACE}(.*)[\:][\:](.*)\(\)%{SPACE}[-]%{SPACE}Exception[\:]%{SPACE}[\\][n]%{JAVACLASS:[@metadata][Exception]}[\:]%{SPACE}%{DATA:message}[\\][n]%{GREEDYDATA:gd}
Here is the log entry
"epayuser1@sp360.com] - EPayApiController::GetFileInformation() - Exception: \nSystem.ArgumentNullException: String reference not set to an instance of a String.\nParameter name: s\n at System.Text.Encoding.GetBytes(String s)\n at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.ToBase64Encode(String plainText) in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\Providers\\ElasticSearch\\ElasticSearchRequest.cs:line 259\n at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.<Search>d__1491.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\Providers\\ElasticSearch\\ElasticSearchRequest.cs:line 118\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\n at Fnf.Agency.EPay.WebApi.DAL.EPayDAL.<GetFileBalances>d__6.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\EPayDAL.cs:line 167\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter
1.GetResult()\n at Fnf.Agency.EPay.WebApi.BLL.EPayBLL.d__5.MoveNext() in C:\src\Agency-Services\EPayment\Dev\Fnf.Agency.EPay.WebApi\Fnf.Agency.EPay.WebApi.BLL\EPayBLL.cs:line 42\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\n at Fnf.Agency.EPay.WebApi.Api.Controllers.EPayApiController.<GetFileInformation>d__3.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.Api\\Controllers\\EpayApiController.cs:line 51
Here is the data returned by the grok debugger:
{
"[@metadata][Exception]": "System.ArgumentNullException",
"message": "String reference not set to an instance of a String.",
"gd": "Parameter name: s\\n at System.Text.Encoding.GetBytes(String s)\\n at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.ToBase64Encode(String plainText) in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.DAL\\\\Providers\\\\ElasticSearch\\\\ElasticSearchRequest.cs:line 259\\n at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.<Search>d__149`1.MoveNext() in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.DAL\\\\Providers\\\\ElasticSearch\\\\ElasticSearchRequest.cs:line 118\\n--- End of stack trace from previous location where exception was thrown ---\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\n at System.Runtime.CompilerServices.TaskAwaiter 1.GetResult()\\n at Fnf.Agency.EPay.WebApi.DAL.EPayDAL.<GetFileBalances>d__6.MoveNext() in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.DAL\\\\EPayDAL.cs:line 167\\n--- End of stack trace from previous location where exception was thrown ---\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\n at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\\n at Fnf.Agency.EPay.WebApi.BLL.EPayBLL.<GetFileInformationByFileNumber>d__5.MoveNext() in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.BLL\\\\EPayBLL.cs:line 42\\n--- End of stack trace from previous location where exception was thrown ---\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\n at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\\n at Fnf.Agency.EPay.WebApi.Api.Controllers.EPayApiController.<GetFileInformation>d__3.MoveNext() in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.Api\\\\Controllers\\\\EpayApiController.cs:line 51}"
Here is the logstash config snippet:
mutate {
add_field => [ "[@metadata][Exception]", "unknown" ]
}
#check if the log entry fits a particular format which woold cause grok to set [@metadata][Exception]
grok {
match => {"messagetext" => "(.*)Error%{SPACE}Type[\:]%{SPACE}%{GREEDYDATA:[@metadata][Exception]}"}
}
#if the previous grok did not parse the log entry, fall into here, I could also have checked the _grokparsefailure but I am modifying a field so why not use its value
if [@metadata][Exception] == "unknown" {
mutate {
#sanity check to ensure I am getting in here
add_field => ["[@metadata][exceptionsearch]", "activated" ]
}
grok {
# This is the problem pattern that works on the grok debugger but fails in logstash
match => {"messagetext" => "(.*)\]%{SPACE}[-]%{SPACE}(.*)[\:][\:](.*)\(\)%{SPACE}[-]%{SPACE}Exception[\:]%{SPACE}[\\][n]%{JAVACLASS:[@metadata][Exception]}[\:]%{SPACE}%{DATA:message}[\\][n]%{GREEDYDATA:gd}"}
}
#The next few lines are just there for debugging purposes
if [@metadata][Exception] == "unknown" {
mutate {
add_field => [ "[@metadata][exceptionsearchsucccess]" , "false" ]
}
} else {
mutate {
add_field => [ "[@metadata][exceptionsearchsucccess]" , "true" ]
}
}
}
Here is the corresponding logstash output:
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_grokparsefailure"
],
"Loglevel" => "ERROR",
"input" => {
"type" => "log"
},
"logtype" => "iis",
"@timestamp" => 2019-07-16T17:41:00.000Z,
"@version" => "1",
"host" => {
"os" => {
"build" => "17763.973",
"name" => "Windows 10 Enterprise",
"family" => "windows",
"version" => "10.0",
"platform" => "windows"
},
"name" => "FNFCORPTECWK014",
"id" => "4be03f54-870e-4081-bdcf-0328bb4c821a",
"architecture" => "x86_64"
},
"beat" => {
"hostname" => "FNFCORPTECWK014",
"name" => "FNFCORPTECWK014",
"version" => "6.7.1"
},
"Messagetext" => "epayuser1@sp360.com] - EPayApiController::GetFileInformation() - Exception: \nSystem.ArgumentNullException: String reference not set to an instance of a String.\nParameter name: s\n at System.Text.Encoding.GetBytes(String s)\n at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.ToBase64Encode(String plainText) in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\Providers\\ElasticSearch\\ElasticSearchRequest.cs:line 259\n at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.<Search>d__149`1.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\Providers\\ElasticSearch\\ElasticSearchRequest.cs:line 118\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\n at Fnf.Agency.EPay.WebApi.DAL.EPayDAL.<GetFileBalances>d__6.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\EPayDAL.cs:line 167\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\n at Fnf.Agency.EPay.WebApi.BLL.EPayBLL.<GetFileInformationByFileNumber>d__5.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.BLL\\EPayBLL.cs:line 42\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\n at Fnf.Agency.EPay.WebApi.Api.Controllers.EPayApiController.<GetFileInformation>d__3.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.Api\\Controllers\\EpayApiController.cs:line 51",
"origtimestamp" => "2019-16-07 13:41:00"
}
{
"offset" => 0,
"Server" => "FNFCAIRVDT042",
"Application_Name" => "Fnf.EPay.WebApi.Api",
"log" => {
"file" => {
"path" => "c:\testlog\Fnf.EPay.WebApi.Api-int-FNFCAIRVDT042-20190716.log"
}
},