How to handle Exceptions

Elastic Stack Logstash
1

Hello Logstash Community,

in my logfiles I have different Exceptions from different Stack Traces. How can I handle singleline and multiline entries within one logfile?

The normal logfile looks like:

<191>Jul 10 13:08:40 FFM30SERVERT0031 DEVTEST2-OO-Logs: Debug,10,metrosystems.mcrm.business.Service.LDAPService,hd0mbbykwvlnidzaxtbquvvc,de,,,LDAP Authentication successful for User 'SUP_A.SCHITZ_US@ASF.ADM.NET'.,,LDAPService,LogEntry

Than I have in the same logfile entries like:

<187>Jul 10 13:09:02 FFM30WEBZST0031 DEVTEST2-OO-Logs: Error,10,MyMetroController,hd0mbbykwvlnidzaxtbquvvc,de,,,Error getting session data for user WoytonGmbH,\"System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://id-pp.metrogroup-networking.com/ecommerce/myMetro/webService/ManageUserAccount.asmx that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.16.207.143:80
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
   --- End of inner exception stack trace ---
Server stack trace: 
   at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
   at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at metrosystems.mcrm.data.DataProvider.Webservice.ManageUserAccount.ManageUserAccountSoap.getAccountByUPN(String UPN)
   at metrosystems.mcrm.data.DataProvider.Webservice.LDAPWebserviceDataProvider.GetAccountByUPN(String loginName) in d:\\TeamCity\\buildAgent\\work\\e9218e867e0db194\\metrosystems.mcrm.data\\DataProvider\\Webservice\\LDAPWebserviceDataProvider.cs:line 55
   at metrosystems.mcrm.data.Repository.LDAPRepository.GetCardholderByLogin(String loginName) in d:\\TeamCity\\buildAgent\\work\\e9218e867e0db194\\metrosystems.mcrm.data\epository\\LDAPRepository.cs:line 80
   at metrosystems.mcrm.oo2.Controllers.MyMetroController.Login(LoginModel loginModel) in d:\\TeamCity\\buildAgent\\work\\e9218e867e0db194\\metrosystems.mcrm.oo2\\Controllers\\MyMetroController.cs:line 213
System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.16.207.143:80
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

For the first part I am using the following filter and it works very good:

filter {
        if [type] == "online_ordering" {
                if !("_grokparsefailure" in [tags]) {
                        grok {
                                match => [ "message", "<%{BASE10NUM:}>%{MONTH:month}%{SPACE}%{MONTHDAY:day}%{SPACE}%{TIME:time}%{SPACE}%{HOSTNAME:server}%{SPACE}%{DATA:logfile}:%{SPACE}%{GREEDYDATA:mesag}" ]
                                remove_field => [ "host" ]
                        }
                        csv {
                                columns => ["level","severity", "logger", "aspnet-sessionid", "organisation", "correlationState", "user", "msg", "exception", "method", "messageType"]
                                separator => ","
                                source => "mesag"
                                remove_field => [ "severity" ]
                        }
                }
        }
}

But how I can also handle the lines which additionally throwing Exceptions?

Does anybody have an idea?

Thanks

Greetings,
ABecker

2

Process all messages with a multiline filter (or codec), then use a conditional to take a peek at the message and decide whether it's a single-line message or if there's a stacktrace that you need to process differently. Or, you might even be able to use the "multiline" tag that I believe the multiline filter adds to all messages that were joined from multiple physical lines.

3

Hello Magnus,

I have created a configuration with a multiline filter. But for some reasons the filter excludes the exception lines.

My filter look like

> filter {
>         if [type] == "online_ordering" {
>                 if !("_grokparsefailure" in [tags]) {
>                         grok {
>                                 match => [ "message", "<%{BASE10NUM:}>%{MONTH:month}%{SPACE}%{MONTHDAY:day}%{SPACE}%{TIME:time}%{SPACE}%{HOSTNAME:server}%{SPACE}%{DATA:logfile}:%{SPACE}%{GREEDYDATA:mesag}" ]
>                                 remove_field => [ "host" ]
>                         }
>                         csv {
>                                 columns => ["level","severity", "logger", "aspnet-sessionid", "organisation", "correlationState", "user", "msg", "exception", "method", "messageType"]
>                                 separator => ","
>                                 source => "mesag"
>                                 remove_field => [ "severity" ]
>                         }
>                         multiline {
>                                 pattern => "^<"
>                                 what => "previous"
>                                 negate => true
>                         }
>                 }
>         }
> }

The normal lines which passes my grok filter will be stored within the elasticsearch db correctly, but the exception part is not present, because every line will be passed as a single line and this single lines do not match to the grok filter. If all the multiline would be stored within the message field, I am sure that the grok filter will work.

Do you have an idea why the multiline filter do not store all lines within the message field?

Greetz
ABecker

4

The multiline filter should be the first filter.

Unrelated to your problem, but

if !("_grokparsefailure" in [tags]) {

can be written like this instead:

if "_grokparsefailure" not in [tags] {
5

I have set the multiline filter as the first filter now.

I also changed the if condition to your suggestion.

But while I have changed the multiline filter nothing happen when I simulate a logfile entry.

Now I am totally confused. Why logstash now is doing nothing?

6

Can my pattern be wrong?

pattern => "^<"

> <187>Jul 10 13:09:02 FFM30WEBZST0031 DEVTEST2-OO-Logs: Error,10,MyMetroController,hd0mbbykwvlnidzaxtbquvvc,de,,,Error getting session data for user WoytonGmbH,\"System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://id-pp.metrogroup-networking.com/ecommerce/myMetro/webService/ManageUserAccount.asmx that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.16.207.143:80
>    at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
>    at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
7

It won't emit a message until it encounters the next start-of-message marker, because until it receives that it can't know whether the current message has ended or if there might be additional lines arriving. In other words, add another logfile entry to the monitored file to have Logstash emit the first one.

8

Hello Magnus,

it seems that my problem with the multiline filter has something to do with my tcp input.

If I use as input file, than the multiline filter will be passed correctly, but if I use tcp input what I am do, nothing happens.

I think that the problem is that each line of the logfile will be send via netcat to the server where logstash is running.

Is it possible to use tcp input together with a multiline filter even if every line will be send via netcat?

So for every line a new connection will be accepted by the client and closed.

Greetz
Abecker

9

If you want this to work it's absolutely crucial that stream_identity is set appropriately so that lines from the same machine can be joined correctly.

But you should join the lines at the source instead, i.e. on the box actually reading each file.

10

Thank you very much Magnus.

Your proposal with stream_identity was the solution for my problem. Now multiline works!

Greetings,
ABecker