Logstash combining multiple entries

prateek_shrivastava · April 17, 2018, 8:25am

My config file looks like:

    input { 
	file {
		path => "path"
		start_position => "beginning"
		codec => multiline {
		pattern => "^\s"
		what => "previous"
		}
	}
} 

filter {
	grok {
      match => [ "message", "(?m)^%{TIMESTAMP_ISO8601:logtime}%{SPACE}\|%{SPACE}\[%{NUMBER:thread_id}\]%{SPACE}\|%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\|%{SPACE}%{GREEDYDATA:classname}%{SPACE}\|%{SPACE}%{GREEDYDATA:logmessage}"]
   }
   
   multiline {
    pattern => "^\s"
    what => "previous"
	negate => false
  }

    date {
        match => ["logtime", "yyyy-MM-dd HH:mm:ss"]
        timezone => "GMT"
   } 
}

output { 
	stdout {
		#codec => "rubydebug"
		codec => line { format => "custom format: %{logtime} %{thread_id} %{loglevel} %{classname} %{logmessage}"}
	} 
}

It is combining multiple entries like:

custom format: 2018-04-17 10:01:32,2018-04-17 09:48:47,2018-04-17 09:48:50 5,25
ERROR BankerbayPL.Global.Application_Error(Global.asax.cs:2976) ,BankerbayPL.Use
rControl.BBNewsfeedControl.getNewsFeed(BBNewsfeedControl.ascx.cs:259) ,Bankerbay
DAL.clsBankerbayDAL.GetPrimaryImagesForDeal(clsBankerbayDAL.cs:25241) Unhandled
,Exception When Fetching Newsfeed url: http://localhost:8080/WebServices/GetNews
Feed data passed: ?newsId=000000&max=2 Exception details : System.Net.WebExcept
ion: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at BankerbayBL.RestClient.MakeRequest(String parameters) in E:.Net v4 Chines
,GetPrimaryImagesForDeal failed. Returned exception : Exception details : System
.Net.WebException: Unable to connect to the remote server ---> System.Net.Socket
s.SocketException: No connection could be made because the target machine active
ly refused it 127.0.0.1:8080
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddre
ss socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Sock
et s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state,
IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at BankerbayBL.RestClient.MakeRequest(String parameters) in E:.Net v4 Chines
e\trunk\BankerbayBL\RestClient.cs:line 101. Params deal_id : 10379

I don't want the entries to be combined. How to make logstash to give separate results for each line?

guyboertje · April 17, 2018, 8:36am

From the info you have supplied it is impossible to help.

We don't know:

what the original data looks like
in the sample you give, where the combined entries problem is
what your ideal output should look like

prateek_shrivastava · April 17, 2018, 9:07am

Hello,

Sorry for providing half information:

My log file looks like:

2018-04-17 10:01:23 | [5] | INFO | BankerbayPL.PostLoginCat.Page_Init(PostLoginCat.Master.cs:80) | pageinit postlogincat Inside 404: session: 15 cookies: System.Web.HttpCookie  
2018-04-17 10:01:23 | [5] | INFO | BankerbayPL.PostLoginCat.Page_Load(PostLoginCat.Master.cs:19) | pageload postlogincat Inside 404: session: 15 cookies: System.Web.HttpCookie  
2018-04-17 10:01:23 | [5] | INFO | BankerbayPL.UserControl.ucGoogleAnalytics.Page_Load(ucGoogleAnalytics.ascx.cs:61) | Testing: ContactUs  
2018-04-17 10:01:28 | [6] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2970) | Unhandled exception : System.Web.HttpException (0x80004005): File does not exist.
   at System.Web.StaticFileHandler.GetFileInfo(String virtualPathWithPathInfo, String physicalPath, HttpResponse response)
   at System.Web.StaticFileHandler.ProcessRequestInternal(HttpContext context, String overrideVirtualPath)
   at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback callback, Object state)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)  
2018-04-17 10:01:28 | [6] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2976) | Unhandled Exception on RequestURL:/images/userProfilePic/Thumb_15_3279_15_09_2017.png  
2018-04-17 10:01:28 | [6] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2978) | Exception on UrlReferrer:http://localhost:3442/StaticPage/ContactUs.aspx  
2018-04-17 10:01:32 | [10] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2970) | Unhandled exception : System.Web.HttpException (0x80004005): The HTTP verb POST used to access path '/webservices/notification/swFailure' is not allowed.
   at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback callback, Object state)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)  
2018-04-17 10:01:32 | [5] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2970) | Unhandled exception : System.Web.HttpException (0x80004005): The HTTP verb POST used to access path '/webservices/notification/swFailure' is not allowed.
   at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback callback, Object state)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)  
2018-04-17 10:01:32 | [10] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2976) | Unhandled Exception on RequestURL:/webservices/notification/swFailure  
2018-04-17 10:01:32 | [5] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2976) | Unhandled Exception on RequestURL:/webservices/notification/swFailure  
2018-04-17 10:01:32 | [10] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2978) | Exception on UrlReferrer:http://localhost:3442/StaticPage/ContactUs.aspx  
2018-04-17 10:01:32 | [5] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2978) | Exception on UrlReferrer:http://localhost:3442/StaticPage/ContactUs.aspx  
2018-04-17 10:01:32 | [5] | ERROR | BankerbayPL.Global.Application_Error(Global.asax.cs:2978) | Exception on UrlReferrer:http://localhost:3442/StaticPage/ContactUs.aspx  ```

The output which I am getting is:

custom format: 2018-04-17 10:01:23 5 INFO BankerbayPL.Global.keepMeLoggedInCheck
(Global.asax.cs:4294)  session user_id value is alive in checkKeepMeLoggedIn end
 : 15
custom format: 2018-04-17 10:01:23 5 INFO BankerbayPL.PostLoginCat.Page_Init(Pos
tLoginCat.Master.cs:80)  pageinit postlogincat Inside 404: session: 15 cookies:
System.Web.HttpCookie
custom format: 2018-04-17 10:01:23 5 INFO BankerbayPL.PostLoginCat.Page_Load(Pos
tLoginCat.Master.cs:19)  pageload postlogincat Inside 404: session: 15 cookies:
System.Web.HttpCookie
custom format: 2018-04-17 10:01:23,2018-04-17 10:01:28 5,6 INFO,ERROR BankerbayP
L.UserControl.ucGoogleAnalytics.Page_Load(ucGoogleAnalytics.ascx.cs:61) ,Bankerb
,Unhandled exception : System.Web.HttpException (0x80004005): File does not exis
t.
   at System.Web.StaticFileHandler.GetFileInfo(String virtualPathWithPathInfo, S
tring physicalPath, HttpResponse response)
   at System.Web.StaticFileHandler.ProcessRequestInternal(HttpContext context, S
tring overrideVirtualPath)
   at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, Asy
ncCallback callback, Object state)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplica
tion.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& compl
etedSynchronously)
custom format: 2018-04-17 10:01:28 6 ERROR BankerbayPL.Global.Application_Error(
Global.asax.cs:2976)  Unhandled Exception on RequestURL:/images/userProfilePic/T
humb_15_3279_15_09_2017.png
custom format: 2018-04-17 10:01:28,2018-04-17 10:01:32 6,10,5 ERROR BankerbayPL.
Global.Application_Error(Global.asax.cs:2978) ,BankerbayPL.Global.Application_Er
ror(Global.asax.cs:2970)  Exception on UrlReferrer:http://localhost:3442/StaticP
,Unhandled exception : System.Web.HttpException (0x80004005): The HTTP verb POST
 used to access path '/webservices/notification/swFailure' is not allowed.
   at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, Asy
ncCallback callback, Object state)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplica
tion.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& compl
etedSynchronously)
custom format: 2018-04-17 10:01:32 10 ERROR BankerbayPL.Global.Application_Error
(Global.asax.cs:2976)  Unhandled Exception on RequestURL:/webservices/notificati
on/swFailure
custom format: 2018-04-17 10:01:32 5 ERROR BankerbayPL.Global.Application_Error(
Global.asax.cs:2976)  Unhandled Exception on RequestURL:/webservices/notificatio
n/swFailure
custom format: 2018-04-17 10:01:32 10 ERROR BankerbayPL.Global.Application_Error
(Global.asax.cs:2978)  Exception on UrlReferrer:http://localhost:3442/StaticPage
/ContactUs.aspx
custom format: 2018-04-17 10:01:32 5 ERROR BankerbayPL.Global.Application_Error(
Global.asax.cs:2978)  Exception on UrlReferrer:http://localhost:3442/StaticPage/
ContactUs.aspx

As you can see in this line :

custom format: 2018-04-17 10:01:23,2018-04-17 10:01:28 5,6 INFO,ERROR.

2 events are combined. I don't want it to be combined. How can I do it?

guyboertje · April 17, 2018, 11:15am

Can you edit your post and paste the source data lines within triple backticks? I need to see whether the lines that start with at System.Web do or don't lead with whitespace.

Also, why are you using the multiline filter?

prateek_shrivastava · April 17, 2018, 11:40am

Hello,

I have edited my post.
I have solved the problem by re-ordering the filters

The correct order should be multiline, grok and then date

I am using multiline filter to combine all the lines in the stacktrace.

guyboertje · April 17, 2018, 11:48am

Good to know the problem is fixed.

The multiline codec is used for combining the stacktrace lines into the main line. It is stream identity aware, meaning that lines from different file paths (stream identity) will be accumulated in separate buffers.

The multiline filter will not, it does not know which file a line (event) comes from and so you can get events from different sources (files) merged into each other.

Unless there is something else going on the multiline codec will be sufficient.

system · May 15, 2018, 11:49am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.