Logstash grok pattern working fine in grok debugger, but fails in logstash conf file to parse

Logstash version - 7.3
Filebeat version - 7.3
OS - Windows

Below is my exception format in application log file-
<log4j:event logger="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" timestamp="1566964971685" level="ERROR" thread="7388">log4j:message

    ======================================
     
     MachineName: PRODMAGWEB1 
     TimeStamp: 8/28/2019 12:02:51 AM 
     FullName: ESM.EasyPurchaseMarketplace.Infrastructure.Logging, Version=2.51.7170.6286, Culture=neutral, PublicKeyToken=null 
     AppDomainName: /LM/W3SVC/8/ROOT-1-132113700121002852 
     WindowsIdentity: ESMSOLUTIONS\EPMServiceAppIdentit 
     Message: Exception occured in PurchaseAPI token logic 

     Guid -  
     UserId -  
     EntityId -  
     ImpersonatedBy -  
     ImpersonatingUserId -  
     ImpersonatedUserName -  


     </log4j:message><log4j:properties><log4j:data name="log4net:UserName" value="ESMSOLUTIONS\EPMServiceAppIdentit" /><log4j:data name="log4jmachinename" value="ProdMagWeb1" /><log4j:data name="log4japp" value="/LM/W3SVC/8/ROOT-1-132113700121002852" /><log4j:data name="log4net:HostName" value="ProdMagWeb1" /></log4j:properties><log4j:throwable><![CDATA[System.Web.HttpException (0x80070057): The remote host closed the connection. The error code is 0x80070057.
       at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
       at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
       at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
       at System.Web.HttpResponse.Flush()
       at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
       at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
       at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
       at Microsoft.AspNet.SignalR.Owin.ServerResponse.Write(ArraySegment`1 data)
       at Microsoft.AspNet.SignalR.Hosting.ResponseExtensions.End(IResponse response, String data)
       at Microsoft.AspNet.SignalR.PersistentConnection.SendJsonResponse(HostContext context, String jsonPayload)
       at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest(HostContext context)
       at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context)
       at Microsoft.AspNet.SignalR.Hubs.HubDispatcher.ProcessRequest(HostContext context)
       at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(IDictionary`2 environment)
       at Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware.Invoke(IOwinContext context)
       at Microsoft.Owin.Infrastructure.OwinMiddlewareTransition.Invoke(IDictionary`2 environment)
       at Microsoft.Owin.Cors.CorsMiddleware.HandleCorsRequestAsync(IOwinContext context, CorsPolicy policy, CorsRequestContext corsRequestContext)
       at Microsoft.Owin.Cors.CorsMiddleware.<Invoke>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Owin.Mapping.MapMiddleware.<Invoke>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at ESM.Purchase.WebApi.Common.GlobalOwinExeptionHandler.<Invoke>d__0.MoveNext()]]></log4j:throwable><log4j:locationInfo class="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" method="WriteToLog" file="" line="0" /></log4j:event>

Grok Pattern that I have written:
.*?: %{WORD:MachineName} .*?\n .*?: %{DATE:timestamp} %{TIME:time} .*?\n .*?: .*?, Version=%{USER:Version}, Culture=%{USER:Culture}, PublicKeyToken=%{USER:PublicKeyToken} .*?\n .*?: %{PATH:AppDomainName}.*?\n .*?: %{WORD:WindowsIdentity}.*?\n .*?: (?<Exception Message>.?*)\n

When I use Grok debugger in Kibana under dev tools, this grok pattern gives me correct results as below-
{
"AppDomainName": "/LM/W3SVC/8/ROOT-1-132113700121002852",
"WindowsIdentity": "ESMSOLUTIONS",
"MachineName": "PRODMAGWEB1",
"Version": "2.51.7170.6286",
"time": "12:02:51",
"PublicKeyToken": "null",
"Exception Message": "Exception occured in PurchaseAPI token logic \r",
"Culture": "neutral",
"timestamp": "8/28/2019"
}

But when is use same grok pattern in logstash config file, it gives grok parse failure.
Below is my logstash config file-

input { 
	beats{
		port=>5044
	}
}

    filter {
          grok {
            match =>{
    		"message" => ".*?: %{WORD:MachineName} .*?\n .*?: %{DATE:timestamp} %{TIME:time} .*?\n .*?: .*?, Version=%{USER:Version}, Culture=%{USER:Culture}, PublicKeyToken=%{USER:PublicKeyToken} .*?\n .*?: %{PATH:AppDomainName}.*?\n .*?: %{WORD:WindowsIdentity}.*?\n .*?: (?<Exception Message>.?*)\n" 
    			}  
    		}
    }

    output {
    if "esm-purchase-api-log" in [tags] { 
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "esm-purchase-api-log-%{+YYYY.MM.dd}"
    }
    }
    }

Can you please tell me where am I going wrong. I am trying this hard since long. Any help would be appreciated!Preformatted text

Please edit your post, select the configuration, and click on </> in the toolbar above the edit pane. You should see the appearance change in the preview pane to the right so that the text

looks like this

Then do the same for the input.

I have edited my post. Can you please check?

If I consume that entire message as a single event using a file input then the grok pattern matches

  "WindowsIdentity" => "ESMSOLUTIONS",
        "timestamp" => "8/28/2019",
    "AppDomainName" => "/LM/W3SVC/8/ROOT-1-132113700121002852",
          "Version" => "2.51.7170.6286",
      "MachineName" => "PRODMAGWEB1",
          "Culture" => "neutral"

etc. So I suspect the problem is the way filebeat is ingesting the event.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.