Multiline configuration

Elastic Stack Beats
1

Hello,

I'm currently trying to get our Filebeat to recognize proper rails logging messages, and I'm using the following configuration for the multiline part:

multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}'
    negate => false
    match => "after"

I know that every line from the Rails logger is prefixed with the timestamp: 2016-07-27T12:00:12.029 6268 for example. So the pattern matches these.
However I'm trying to get EVERY line that comes after this to be appended to the first match, unless a new timestamp is detected.
an Entry example:

2016-07-27T12:00:13.485 6268 [DEBUG] <?xml version="1.0" encoding="UTF-8"?>
  <soapenv:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wsp="http://www.dnb.com/DNB_WebServices/Providers/GetNotifications_V5/wsp_GetNotifications_V5" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:http="http://schemas.xmlsoap.org/wsdl/http/" xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:soap12="http://schemas.xmlsoap.org/wsdl/soap12/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/">
<soapenv:Body>
<wsp:ws_GetNotifications>
  <getNotificationsRequest>
    <UserId>***FILTERED***</UserId>
    <Password>***FILTERED***</Password>
    <getNotificationsRequestData>
      <Product_Name>Enterprise Management</Product_Name>
      <Notification_Level>Level 2</Notification_Level>
      <Detection_Date_From>2016-07-27T09:00:12+02:00</Detection_Date_From>
      <Detection_Date_To>2016-07-27T12:00:12+02:00</Detection_Date_To>
    </getNotificationsRequestData>
  </getNotificationsRequest>
</wsp:ws_GetNotifications>

</soapenv:Body>
</soapenv:Envelope>

But I'm still getting the XML as single lines inside Kibana, am I missing something in the multiline config?
Something else I found in my log files:

2016-08-08T09:52:42.707 1676 [INFO] Started GET "/" for 82.135.30.177 at 2016-08-08 09:52:42 +0200
2016-08-08T09:52:42.707 1676 [INFO] Processing by DashboardsController#show as HTML
2016-08-08T09:52:42.718 1676 [INFO] Redirected to https://playground.riskmethods.net/users/login
2016-08-08T09:52:42.718 1676 [INFO] Completed 302 Found in 11ms (ActiveRecord: 0.0ms)
2016-08-08T09:52:42.754 1685 [INFO] Started GET "/users/login" for 82.135.30.177 at 2016-08-08 09:52:42 +0200
2016-08-08T09:52:42.755 1685 [INFO] Processing by SessionsController#new as HTML
2016-08-08T09:52:42.762 1685 [INFO]   Rendered admins/shared/_links.erb (1.0ms)
2016-08-08T09:52:42.764 1685 [INFO]   Rendered shared/_csm_contact.html.erb (0.7ms)
2016-08-08T09:52:42.764 1685 [INFO]   Rendered users/sessions/new.html.erb within layouts/general (6.3ms)
2016-08-08T09:52:42.771 1685 [INFO]   Rendered layouts/application.html.erb (6.1ms)
2016-08-08T09:52:42.789 1685 [INFO] Completed 200 OK in 34ms (Views: 14.7ms | ActiveRecord: 13.7ms)

Why is it concatinating all those in a single message?

2

Try to test your multiline pattern with this Go playground: https://www.elastic.co/guide/en/beats/filebeat/1.2/multiline-examples.html#_testing_your_regexp_pattern_for_multiline

3

I did, and the result I get there is not the result I'm seeing in my Kibana output.
The matching is working on this playground thing, but the aggregation on how it's submitted is something this playground does not do.

Or my Kibana is stubborn

4

Can you enabled the file output and check if the output in the file is in one event or multiple events? Alternatively you can run filebeat with -e -d "publish" to see the events.

BTW: Which version of filebeat are you using?

5

latest version.
Finally got it working, by setting negate to true, and surrounding my string values with '

6

I have been following this thread for a few days. I was having a hell of a time getting the FileBeats pattern to work with Logstash. I discovered that for some reason the ^ character in the pattern isn't keying on the first character in a line.... but rather the first character in a file. Which would seem to be a ridiculous implementation of a new line regex character.

Am I missing something? I stopped using the ^ and now I am getting logs to group together (success!)... but now there are several stack traces being appended to the same message.

Is this a known behavior? Or am I doing something wrong?

I am using SLF4J logs with the log4j input type in FileBeats.

7

@pg99bassists Please consider creating a new topic, as this one has been marked as solved already. The regex is applied to every single line in multiline. That is ^ should do the right thing. When doing regexes on already concatenated multiline events it depends on the regex engine how ^ will actually be handled. It's not clear to me what exactly you're doing and where you apply regexes. Please add configs + filebeat/logstash versions you're using.

8

This topic was automatically closed after 21 days. New replies are no longer allowed.