Split multiple lines (using Grok filter) in message so they display as newlines in Kibana

[Warren]

So i have a message in Kibana and it looks like this:

[2018-06-07 11:02:33] testing.ERROR: Class 'App\Http\Controllers\Deal' not found {"exception":"[object] (Symfony\Component\Debug\Exception\FatalThrowableError(code: 0): Class 'App\Http\Controllers\Deal' not found at /app/Http/Controllers/TestingSyslogController.php:13)\n[stacktrace]\n#0 [internal function]: App\Http\Controllers\TestingSyslogController->index()\n#1 /vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): call_user_func_array(Array, Array)\n#2 /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\Routing\Controller->callAction('index', Array)\n#3 /vendor/laravel/framework/src/Illuminate/Routing/Route.php(212): Illuminate\Routing\ControllerDispatcher->dispatch(Object(Illuminate\Routing\Route), Object(App\Http\Controllers\TestingSyslogController), 'index')\n#4 /vendor/laravel/framework/src/Illuminate/Routing/Route.php(169): Illuminate\Routing\Route->runController()\n#5

How do I display the message line by line in a single message field e.g. use the \n to display the message in kibana like this

[2018-06-07 11:02:33] testing.ERROR: Class 'App\Http\Controllers\Deal' not found {"exception":"[object] (Symfony\Component\Debug\Exception\FatalThrowableError(code: 0): Class 'App\Http\Controllers\Deal' not found at /app/Http/Controllers/TestingSyslogController.php:13)\n[stacktrace]\n

0 [internal function]: App\Http\Controllers\TestingSyslogController->index()\n

1 /vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): call_user_func_array(Array, Array)\n

2 /vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\Routing\Controller->callAction('index', Array)\n

3 /vendor/laravel/framework/src/Illuminate/Routing/Route.php(212): Illuminate\Routing\ControllerDispatcher->dispatch(Object(Illuminate\Routing\Route), Object(App\Http\Controllers\TestingSyslogController), 'index')\n

4 /vendor/laravel/framework/src/Illuminate/Routing/Route.php(169): Illuminate\Routing\Route->runController()\n#5

I'm trying a grok filter... and the debugger at http://grokdebug.herokuapp.com/ is very useful.

I've written this:
%{DATESTAMP:timestamp}%{SPACE}]%{SPACE}%{NOTSPACE:instance}%{SPACE}(?(.|\r|\n)*)

but it still shows a single line...

This regex would match the line break.. \[a-z]{1}
and this rexeg should match the line break and line number \[a-z]{1}(?:#)\d{1}

but neither of these match

%{DATESTAMP:timestamp}%{SPACE}]%{SPACE}%{NOTSPACE:instance}%{SPACE}(?\[a-z]1})

%{DATESTAMP:timestamp}%{SPACE}]%{SPACE}%{NOTSPACE:instance}%{SPACE}(?\[a-z]{1}(?:#)\d{1,2})

[Carmageddon]

I've had the exactly same issue you did the last two days.
Until I stepped back, and thought "why the hell are we still using GROK filter and regex to break it down?"

Turns out it is no longer needed, it was carry over from legacy system that has been decomissioned.
I tested the same message structured in a JSON format instead - works perfectly no need to break it down into fields using any GROK regex.
Attached example how it looks like in Kibana:

JSON looks like:

{.... ,"failureReason": "java.lang.AssertionError: expected [35] but found [25.]\n\tat org.testng.Assert.fail(Assert.java:94)\n......",}

Please note that I lost 2 hours until I realized each ROW in logfile should be a JSON object {...}.
You cant send 2+ objects in an array like [{..},{..}] - filebeat cant unmarshal that.

Hope that helps, Good luck!

[Warren]

Hi,
Thanks for replying.
So I don't need this startmsg.regex in my rsyslog conf file? but then they will be all separate rsyslog entries? Do I need to use multiline filter?

input(type="imfile"
File="/home/app/2018-06-22.log"
startmsg.regex="[[:digit:]]{4}-[[:digit:]]{1,2}-[[:digit:]]{1,2} [[:digit:]]{1,2}:[[:digit:]]{1,2}:[[:digit:]]{1,2}"
)

This is my current Rsyslog JSON template...

template(name="json-template"
type="list") {
constant(value="{")
constant(value=""@timestamp":"") property(name="timereported" dateFormat="rfc3339")
constant(value="","@version":"1")
constant(value="","message":"") property(name="msg" format="json")
constant(value="","sysloghost":"") property(name="hostname")
constant(value="","severity":"") property(name="syslogseverity-text")
constant(value="","facility":"") property(name="syslogfacility-text")
constant(value="","programname":"") property(name="programname")
constant(value="","procid":"") property(name="procid")
constant(value=""}\n")
}

[Carmageddon]

Let me clarify - the input log is controlled by me, so I am easily able to modify it to output JSON instead of the log line like yours, and this solves the problem for me. I dont know anything about Rsyslog configuration.

If you cant modify your input to place JSON, perhaps you can manipulate the input and write a JSON into another file, and only monitor/submit to Logstash only that processed file? You wont have to deal with GROK then.