Logstash kv filter takes each line of log as separate log

Elastic Stack Logstash
1

Logstash version - 7.3
Filebeat version - 7.3
OS - Windows

My log format-
<log4j:event logger="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" timestamp="1566964971685" level="ERROR" thread="7388">log4j:message

======================================
 
 MachineName: PRODMAGWEB1 
 TimeStamp: 8/28/2019 12:02:51 AM 
 FullName: ESM.EasyPurchaseMarketplace.Infrastructure.Logging, Version=2.51.7170.6286, Culture=neutral, PublicKeyToken=null 
 AppDomainName: /LM/W3SVC/8/ROOT-1-132113700121002852 
 WindowsIdentity: ESMSOLUTIONS\EPMServiceAppIdentit 
 Message: Exception occured in PurchaseAPI token logic 

 Guid -  
 UserId -  
 EntityId -  
 ImpersonatedBy -  
 ImpersonatingUserId -  
 ImpersonatedUserName -  


</log4j:message><log4j:properties><log4j:data name="log4net:UserName" value="ESMSOLUTIONS\EPMServiceAppIdentit" /><log4j:data name="log4jmachinename" value="ProdMagWeb1" /><log4j:data name="log4japp" value="/LM/W3SVC/8/ROOT-1-132113700121002852" /><log4j:data name="log4net:HostName" value="ProdMagWeb1" /></log4j:properties><log4j:throwable><![CDATA[System.Web.HttpException (0x80070057): The remote host closed the connection. The error code is 0x80070057.
   at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
   at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
   at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.AspNet.SignalR.Owin.ServerResponse.Write(ArraySegment`1 data)
   at Microsoft.AspNet.SignalR.Hosting.ResponseExtensions.End(IResponse response, String data)
   at Microsoft.AspNet.SignalR.PersistentConnection.SendJsonResponse(HostContext context, String jsonPayload)
   at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest(HostContext context)
   at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context)
   at Microsoft.AspNet.SignalR.Hubs.HubDispatcher.ProcessRequest(HostContext context)
   at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(IDictionary`2 environment)
   at Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware.Invoke(IOwinContext context)
   at Microsoft.Owin.Infrastructure.OwinMiddlewareTransition.Invoke(IDictionary`2 environment)
   at Microsoft.Owin.Cors.CorsMiddleware.HandleCorsRequestAsync(IOwinContext context, CorsPolicy policy, CorsRequestContext corsRequestContext)
   at Microsoft.Owin.Cors.CorsMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Mapping.MapMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ESM.Purchase.WebApi.Common.GlobalOwinExeptionHandler.<Invoke>d__0.MoveNext()]]></log4j:throwable><log4j:locationInfo class="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" method="WriteToLog" file="" line="0" /></log4j:event>

Logstash conf file-

input { 
	beats{
		port=>5044
	}
}

filter { 
	kv {
	 value_split => ":"
	}
}

output {
if "esm-purchase-api-log" in [tags] { 
elasticsearch {
hosts => ["localhost:9200"]
index => "esm-purchase-api-log-%{+YYYY.MM.dd}"
}
}
}

It is splitting fields correctly based on colon( : ) but it takes each line as separate log as shown in screenshot-

image
image.png1140×293 21.5 KB

It should take all values- MachineName,FullName,Timestamp,AppDomainName and WindowsIdentity into one log but it is taking each line as separate log entry.

What should I do to so that it is considered as 1 single log?

2

Have filebeat manage the multiline messages.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.