How to use Multiline

Elastic Stack Beats
1

I am trying to use the new Multiline option in filebeat 1.2 but is having no success. I would like to take the logs that Ola Hallengren's (https://ola.hallengren.com) scripts makes. I have an example of one below. What I am trying to do is to have each line that starts with "Date & Time" become one line until the next "Date & Time" but it keeps giving me the error:
ERR Stop Harvesting. Unexpected encoding line reader error: unknow matcher type:

Date and time: 2015-11-29 01:00:01
Server: SQL1
Version: 10.50.2550.0
Edition: Standard Edition (64-bit)
Procedure: [SQLMaintenance].[dbo].[DatabaseIntegrityCheck]
Parameters: @Databases = 'SYSTEM_DATABASES', @CheckCommands = 'CHECKDB', @PhysicalOnly = 'N', @NoIndex = 'N', @ExtendedLogicalChecks = 'N', @TabLock = 'N', @FileGroups = NULL, @Objects = NULL, @LockTimeout = NULL, @LogToTable = 'Y', @Execute = 'Y'
Source: https://ola.hallengren.com
 
Date and time: 2015-11-29 01:00:01
Database: [master]
Status: ONLINE
Standby: No
Updateability: READ_WRITE
User access: MULTI_USER
Is accessible: Yes
Recovery model: SIMPLE
 
Date and time: 2015-11-29 01:00:01
Command: DBCC CHECKDB ([master]) WITH NO_INFOMSGS, ALL_ERRORMSGS, DATA_PURITY
Outcome: Succeeded
Duration: 00:00:13
Date and time: 2015-11-29 01:00:14
 
Date and time: 2015-11-29 01:00:14
Database: [model]
Status: ONLINE
Standby: No
Updateability: READ_WRITE
User access: MULTI_USER
Is accessible: Yes
Recovery model: SIMPLE
 
Date and time: 2015-11-29 01:00:14
Command: DBCC CHECKDB ([model]) WITH NO_INFOMSGS, ALL_ERRORMSGS, DATA_PURITY
Outcome: Succeeded
Duration: 00:00:00
Date and time: 2015-11-29 01:00:14
 
Date and time: 2015-11-29 01:00:14
Database: [msdb]
Status: ONLINE
Standby: No
Updateability: READ_WRITE
User access: MULTI_USER
Is accessible: Yes
Recovery model: SIMPLE
 
Date and time: 2015-11-29 01:00:14
Command: DBCC CHECKDB ([msdb]) WITH NO_INFOMSGS, ALL_ERRORMSGS, DATA_PURITY
Outcome: Succeeded
Duration: 00:00:04
Date and time: 2015-11-29 01:00:18
 
Date and time: 2015-11-29 01:00:18

Here is my filebeat.yml for the Multiline part:

 # Mutiline can be used for log messages spanning multiple lines. This is common
      # for Java Stack Traces or C-Line Continuation
      multiline:

        # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
        pattern: ^\Date and time

        # Defines if the pattern set under pattern should be negated or not. Default is false.
        #negate: false

        # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
        # that was (not) matched before or after or as long as a pattern is not matched based on negate.
        # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
        #match: after

        # The maximum number of lines that are combined to one event.
        # In case there are more the max_lines the additional lines are discarded.
        # Default is 500
        #max_lines: 500

        # After the defined timeout, an multiline event is sent even if no new pattern was found to start a new event
        # Default is 5s.
        #timeout: 5s

If i don't use multiline I am getting output but as soon as I try to use it I get the error above.

2

multiline is comparing only subsequent lines. The pattern can not work. Plus, match MUST be set to before or after. See error message complaining about unknown matcher type.

with logs always having an empty line, maybe something like this works:

multiline:
  pattern: "^$"
  negate: true
  match: before

using this filter, whenever the previous line was not empty, the current line is merged into the multiline event.

3

@Aron_Pedersen @steffens I am using ubuntu 14.04 LTS and when I try to update my filebeat by typing "sudo apt-get install filebeat " terminal tells me that "filebeat is already the newest version."
So how come you talking about filbeat 1.2 ?? Are you guys working on some other OS for which recent filebeat version has been released ??

4

1.1 and 1.2 are available as nightly builds. 1.2 is currently build from master and the number 1.2 does not really indicate this becoming the next version.

https://beats-nightlies.s3.amazonaws.com/index.html?prefix=

Nightlies are development snapshots and might fail. The next version to be released is 1.1

5

@steffens So by what time I can get the functionality of codec multiline for filebeat so that i can take care of stack traces ????