Pattern works on the grok debugger but fails in logstash

Can someone please tell me how I can debug this pattern and corresponding log entry that works on the grok debugger but fails in logstash

Here is the code that works on the grok debugger:

(.*)\]%{SPACE}[-]%{SPACE}(.*)[\:][\:](.*)\(\)%{SPACE}[-]%{SPACE}Exception[\:]%{SPACE}[\\][n]%{JAVACLASS:[@metadata][Exception]}[\:]%{SPACE}%{DATA:message}[\\][n]%{GREEDYDATA:gd}

Here is the log entry

"epayuser1@sp360.com] - EPayApiController::GetFileInformation() - Exception: \nSystem.ArgumentNullException: String reference not set to an instance of a String.\nParameter name: s\n at System.Text.Encoding.GetBytes(String s)\n at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.ToBase64Encode(String plainText) in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\Providers\\ElasticSearch\\ElasticSearchRequest.cs:line 259\n at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.<Search>d__1491.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\Providers\\ElasticSearch\\ElasticSearchRequest.cs:line 118\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\n at Fnf.Agency.EPay.WebApi.DAL.EPayDAL.<GetFileBalances>d__6.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\EPayDAL.cs:line 167\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\n at Fnf.Agency.EPay.WebApi.BLL.EPayBLL.d__5.MoveNext() in C:\src\Agency-Services\EPayment\Dev\Fnf.Agency.EPay.WebApi\Fnf.Agency.EPay.WebApi.BLL\EPayBLL.cs:line 42\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\n at Fnf.Agency.EPay.WebApi.Api.Controllers.EPayApiController.<GetFileInformation>d__3.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.Api\\Controllers\\EpayApiController.cs:line 51

Here is the data returned by the grok debugger:

{
  "[@metadata][Exception]": "System.ArgumentNullException",
  "message": "String reference not set to an instance of a String.",
  "gd": "Parameter name: s\\n   at System.Text.Encoding.GetBytes(String s)\\n   at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.ToBase64Encode(String plainText) in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.DAL\\\\Providers\\\\ElasticSearch\\\\ElasticSearchRequest.cs:line 259\\n   at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.<Search>d__149`1.MoveNext() in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.DAL\\\\Providers\\\\ElasticSearch\\\\ElasticSearchRequest.cs:line 118\\n--- End of stack trace from previous location where exception was thrown ---\\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\n   at System.Runtime.CompilerServices.TaskAwaiter 1.GetResult()\\n   at Fnf.Agency.EPay.WebApi.DAL.EPayDAL.<GetFileBalances>d__6.MoveNext() in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.DAL\\\\EPayDAL.cs:line 167\\n--- End of stack trace from previous location where exception was thrown ---\\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\n   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\\n   at Fnf.Agency.EPay.WebApi.BLL.EPayBLL.<GetFileInformationByFileNumber>d__5.MoveNext() in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.BLL\\\\EPayBLL.cs:line 42\\n--- End of stack trace from previous location where exception was thrown ---\\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\n   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\\n   at Fnf.Agency.EPay.WebApi.Api.Controllers.EPayApiController.<GetFileInformation>d__3.MoveNext() in C:\\\\src\\\\Agency-Services\\\\EPayment\\\\Dev\\\\Fnf.Agency.EPay.WebApi\\\\Fnf.Agency.EPay.WebApi.Api\\\\Controllers\\\\EpayApiController.cs:line 51}"

Here is the logstash config snippet:

mutate {
	add_field => [ "[@metadata][Exception]", "unknown" ]
}
#check if the log entry fits a particular format which woold cause grok to set [@metadata][Exception]
grok {
	match => {"messagetext" => "(.*)Error%{SPACE}Type[\:]%{SPACE}%{GREEDYDATA:[@metadata][Exception]}"}
}
#if the previous grok did not parse the log entry, fall into here, I could also have checked the _grokparsefailure but I am modifying a field so why not use its value
if [@metadata][Exception] == "unknown" {
    mutate {
	    #sanity check to ensure I am getting in here
	    add_field => ["[@metadata][exceptionsearch]", "activated" ]
	}
	grok {
	    # This is the problem pattern that works on the grok debugger but fails in logstash
		match => {"messagetext" => "(.*)\]%{SPACE}[-]%{SPACE}(.*)[\:][\:](.*)\(\)%{SPACE}[-]%{SPACE}Exception[\:]%{SPACE}[\\][n]%{JAVACLASS:[@metadata][Exception]}[\:]%{SPACE}%{DATA:message}[\\][n]%{GREEDYDATA:gd}"}
	}
	#The next few lines are just there for debugging purposes
	if [@metadata][Exception] == "unknown" {
	    mutate {
			add_field => [ "[@metadata][exceptionsearchsucccess]" , "false" ]
    	}
    } else {
        mutate {
    		add_field => [ "[@metadata][exceptionsearchsucccess]" , "true" ]
    	}
    			
    }
}

Here is the corresponding logstash output:

   "tags" => [
            [0] "beats_input_codec_plain_applied",
            [1] "_grokparsefailure"
       ],
       "Loglevel" => "ERROR",
           "input" => {
                   "type" => "log"
          },
         "logtype" => "iis",
          "@timestamp" => 2019-07-16T17:41:00.000Z,
          "@version" => "1",
           "host" => {
            "os" => {
                  "build" => "17763.973",
                  "name" => "Windows 10 Enterprise",
                   "family" => "windows",
                    "version" => "10.0",
                    "platform" => "windows"
              },
             "name" => "FNFCORPTECWK014",
              "id" => "4be03f54-870e-4081-bdcf-0328bb4c821a",
               "architecture" => "x86_64"
      },
       "beat" => {
        "hostname" => "FNFCORPTECWK014",
         "name" => "FNFCORPTECWK014",
          "version" => "6.7.1"
},
"Messagetext" => "epayuser1@sp360.com] - EPayApiController::GetFileInformation() - Exception: \nSystem.ArgumentNullException: String reference not set to an instance of a String.\nParameter name: s\n   at System.Text.Encoding.GetBytes(String s)\n   at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.ToBase64Encode(String plainText) in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\Providers\\ElasticSearch\\ElasticSearchRequest.cs:line 259\n   at Fnf.Agency.EPay.WebApi.DAL.Providers.ElasticSearch.ElasticSearchRequest.<Search>d__149`1.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\Providers\\ElasticSearch\\ElasticSearchRequest.cs:line 118\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\n   at Fnf.Agency.EPay.WebApi.DAL.EPayDAL.<GetFileBalances>d__6.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.DAL\\EPayDAL.cs:line 167\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\n   at Fnf.Agency.EPay.WebApi.BLL.EPayBLL.<GetFileInformationByFileNumber>d__5.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.BLL\\EPayBLL.cs:line 42\n--- End of stack trace from previous location where exception was thrown ---\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\n   at Fnf.Agency.EPay.WebApi.Api.Controllers.EPayApiController.<GetFileInformation>d__3.MoveNext() in C:\\src\\Agency-Services\\EPayment\\Dev\\Fnf.Agency.EPay.WebApi\\Fnf.Agency.EPay.WebApi.Api\\Controllers\\EpayApiController.cs:line 51",
   "origtimestamp" => "2019-16-07 13:41:00"

}
{
"offset" => 0,
"Server" => "FNFCAIRVDT042",
"Application_Name" => "Fnf.EPay.WebApi.Api",
"log" => {
"file" => {
"path" => "c:\testlog\Fnf.EPay.WebApi.Api-int-FNFCAIRVDT042-20190716.log"
}
},

By the way, the first component of the pattern is (.*) but for some reason it did not come over when I a pasted it Ok I used the code quote and format came the way it originally was.

(.*)\]%{SPACE}[-]%{SPACE}(.*)[\:][\:](.*)\(\)%{SPACE}[-]%{SPACE}Exception[\:]%{SPACE}[\\][n]%{JAVACLASS:[@metadata][Exception]}[\:]%{SPACE}%{DATA:message}[\\][n]%{GREEDYDATA:gd}

You really need to learn enough markdown to format your posts if you expect folks to read them. Specifically code quoting. A few minutes work that will pay many dividends.

I could probably do more on the markdown side. However, I was hoping to get a quick sanity check on the base topic of the glaring discrepancy between the Grok debugger and Logstash giving frustratingly different results on the same input. Granted there may be more than meets the eye and some formatting may be occurring as you paste into the Grok input fields that may make the input pass the debugger's parser but this topic has been discussed enough times for you folks to provide some insights as to what to avoid, what to look for. After all 2 tools from the same application are at odds with each other and that can't be good. Not good at all.

I formatted it with the pre-fornatted code markdown.

I'll let someone from the elastic team address that.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.