Multiline parsing patterns

[Raghuveer_SJ (Raghuveer)]

I want to parse a standard JAVA exception which looks like :

2018-09-04 05:29:03.955 [default task-38] ERROR c.r.e.u.util.MongoConnectionUtil.createMongoUser - Exception occured while creating mongo userCommand failed with error 11000: 'User "asdf" already exists' on server 192.168.1.33:27017. The full response is { "ok" : 0.0, "errmsg" : "User \"asdf\" already exists", "code" : 11000, "codeName" : "DuplicateKey" }
com.mongodb.MongoCommandException: Command failed with error 11000: 'User "qwer" already exists' on server 192.168.1.33:27017. The full response is { "ok" : 0.0, "errmsg" : "User \"asdf\" already exists", "code" : 11000, "codeName" : "DuplicateKey" }
	at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
	at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114)

My filebeat.yml has the configuration :

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - C:\logs\test.log
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:'
  multiline.negate: false
  multiline.match: after

and my logstash.conf input looks like :

input {

beats {
	port=>5044
        codec => multiline {
               pattern => "^\s"
              what => "previous"
}
}

But logstash says failed to parse the pattern. If I remove the codec configuration then only the first line of the exception is getting parsed. Kindly help.

[kvch (Noémi Ványi)]

What do you want to aggregate in Logstash? Filebeat already takes care of aggregating multiple lines into a single event. There is no need for further aggregation.

[Raghuveer_SJ (Raghuveer)]

ok, so let say I removed the logstash codec configuration. So whatever I posted for filebeat is that enough for aggregation of logs for JAVA exception stackstrace ? Is the configuration correct ? Because I don't see it happening.

[kvch (Noémi Ványi)]

Yes, the log you've pasted here differs from the examples here: https://www.elastic.co/guide/en/beats/filebeat/current/_examples_of_multiline_configuration.html#_java_stack_traces

You could try to match for the beginning of the logs which is a timestamp.

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after

[Raghuveer_SJ (Raghuveer)]

So this means log starting with a timestamp and anything after that should be grouped till you see a log with another timestamp right ? (I am pointing to caused by text that will be there in exceptions)