Logstash version - 7.3
Filebeat version - 7.3
OS - Windows
Below is my exception format in application log file-
<log4j:event logger="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" timestamp="1566964971685" level="ERROR" thread="7388">log4j:message
======================================
MachineName: PRODMAGWEB1
TimeStamp: 8/28/2019 12:02:51 AM
FullName: ESM.EasyPurchaseMarketplace.Infrastructure.Logging, Version=2.51.7170.6286, Culture=neutral, PublicKeyToken=null
AppDomainName: /LM/W3SVC/8/ROOT-1-132113700121002852
WindowsIdentity: ESMSOLUTIONS\EPMServiceAppIdentit
Message: Exception occured in PurchaseAPI token logic
Guid -
UserId -
EntityId -
ImpersonatedBy -
ImpersonatingUserId -
ImpersonatedUserName -
</log4j:message><log4j:properties><log4j:data name="log4net:UserName" value="ESMSOLUTIONS\EPMServiceAppIdentit" /><log4j:data name="log4jmachinename" value="ProdMagWeb1" /><log4j:data name="log4japp" value="/LM/W3SVC/8/ROOT-1-132113700121002852" /><log4j:data name="log4net:HostName" value="ProdMagWeb1" /></log4j:properties><log4j:throwable><![CDATA[System.Web.HttpException (0x80070057): The remote host closed the connection. The error code is 0x80070057.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpResponse.Flush()
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Microsoft.AspNet.SignalR.Owin.ServerResponse.Write(ArraySegment`1 data)
at Microsoft.AspNet.SignalR.Hosting.ResponseExtensions.End(IResponse response, String data)
at Microsoft.AspNet.SignalR.PersistentConnection.SendJsonResponse(HostContext context, String jsonPayload)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest(HostContext context)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context)
at Microsoft.AspNet.SignalR.Hubs.HubDispatcher.ProcessRequest(HostContext context)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(IDictionary`2 environment)
at Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware.Invoke(IOwinContext context)
at Microsoft.Owin.Infrastructure.OwinMiddlewareTransition.Invoke(IDictionary`2 environment)
at Microsoft.Owin.Cors.CorsMiddleware.HandleCorsRequestAsync(IOwinContext context, CorsPolicy policy, CorsRequestContext corsRequestContext)
at Microsoft.Owin.Cors.CorsMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Mapping.MapMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ESM.Purchase.WebApi.Common.GlobalOwinExeptionHandler.<Invoke>d__0.MoveNext()]]></log4j:throwable><log4j:locationInfo class="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" method="WriteToLog" file="" line="0" /></log4j:event>
Grok Pattern that I have written:
.*?: %{WORD:MachineName} .*?\n .*?: %{DATE:timestamp} %{TIME:time} .*?\n .*?: .*?, Version=%{USER:Version}, Culture=%{USER:Culture}, PublicKeyToken=%{USER:PublicKeyToken} .*?\n .*?: %{PATH:AppDomainName}.*?\n .*?: %{WORD:WindowsIdentity}.*?\n .*?: (?<Exception Message>.?*)\n
When I use Grok debugger in Kibana under dev tools, this grok pattern gives me correct results as below-
{
"AppDomainName": "/LM/W3SVC/8/ROOT-1-132113700121002852",
"WindowsIdentity": "ESMSOLUTIONS",
"MachineName": "PRODMAGWEB1",
"Version": "2.51.7170.6286",
"time": "12:02:51",
"PublicKeyToken": "null",
"Exception Message": "Exception occured in PurchaseAPI token logic \r",
"Culture": "neutral",
"timestamp": "8/28/2019"
}
But when is use same grok pattern in logstash config file, it gives grok parse failure.
Below is my logstash config file-
input {
beats{
port=>5044
}
}
filter {
grok {
match =>{
"message" => ".*?: %{WORD:MachineName} .*?\n .*?: %{DATE:timestamp} %{TIME:time} .*?\n .*?: .*?, Version=%{USER:Version}, Culture=%{USER:Culture}, PublicKeyToken=%{USER:PublicKeyToken} .*?\n .*?: %{PATH:AppDomainName}.*?\n .*?: %{WORD:WindowsIdentity}.*?\n .*?: (?<Exception Message>.?*)\n"
}
}
}
output {
if "esm-purchase-api-log" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "esm-purchase-api-log-%{+YYYY.MM.dd}"
}
}
}
Can you please tell me where am I going wrong. I am trying this hard since long. Any help would be appreciated!Preformatted text