Logstash version - 7.3
Filebeat version - 7.3
OS - Windows
My log format-
<log4j:event logger="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" timestamp="1566964971685" level="ERROR" thread="7388">log4j:message
======================================
MachineName: PRODMAGWEB1
TimeStamp: 8/28/2019 12:02:51 AM
FullName: ESM.EasyPurchaseMarketplace.Infrastructure.Logging, Version=2.51.7170.6286, Culture=neutral, PublicKeyToken=null
AppDomainName: /LM/W3SVC/8/ROOT-1-132113700121002852
WindowsIdentity: ESMSOLUTIONS\EPMServiceAppIdentit
Message: Exception occured in PurchaseAPI token logic
Guid -
UserId -
EntityId -
ImpersonatedBy -
ImpersonatingUserId -
ImpersonatedUserName -
</log4j:message><log4j:properties><log4j:data name="log4net:UserName" value="ESMSOLUTIONS\EPMServiceAppIdentit" /><log4j:data name="log4jmachinename" value="ProdMagWeb1" /><log4j:data name="log4japp" value="/LM/W3SVC/8/ROOT-1-132113700121002852" /><log4j:data name="log4net:HostName" value="ProdMagWeb1" /></log4j:properties><log4j:throwable><![CDATA[System.Web.HttpException (0x80070057): The remote host closed the connection. The error code is 0x80070057.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpResponse.Flush()
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Microsoft.AspNet.SignalR.Owin.ServerResponse.Write(ArraySegment`1 data)
at Microsoft.AspNet.SignalR.Hosting.ResponseExtensions.End(IResponse response, String data)
at Microsoft.AspNet.SignalR.PersistentConnection.SendJsonResponse(HostContext context, String jsonPayload)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest(HostContext context)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context)
at Microsoft.AspNet.SignalR.Hubs.HubDispatcher.ProcessRequest(HostContext context)
at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(IDictionary`2 environment)
at Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware.Invoke(IOwinContext context)
at Microsoft.Owin.Infrastructure.OwinMiddlewareTransition.Invoke(IDictionary`2 environment)
at Microsoft.Owin.Cors.CorsMiddleware.HandleCorsRequestAsync(IOwinContext context, CorsPolicy policy, CorsRequestContext corsRequestContext)
at Microsoft.Owin.Cors.CorsMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Mapping.MapMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at ESM.Purchase.WebApi.Common.GlobalOwinExeptionHandler.<Invoke>d__0.MoveNext()]]></log4j:throwable><log4j:locationInfo class="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" method="WriteToLog" file="" line="0" /></log4j:event>
Logstash conf file-
input {
beats{
port=>5044
}
}
filter {
kv {
value_split => ":"
}
}
output {
if "esm-purchase-api-log" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "esm-purchase-api-log-%{+YYYY.MM.dd}"
}
}
}
It is splitting fields correctly based on colon( : ) but it takes each line as separate log as shown in screenshot-
It should take all values- MachineName,FullName,Timestamp,AppDomainName and WindowsIdentity into one log but it is taking each line as separate log entry.
What should I do to so that it is considered as 1 single log?