Beats with multiline

Elastic Stack Beats
1

can i use FileBeat With multiline?

2

Not yet, but we are working on an implementation. Best is to follow this issue: https://github.com/elastic/filebeat/issues/301

3

@ruflin by what time it will get available ??? We need this as we need to take care of the laravel logs stack traces.

4

We currently plan to release 1.1 with this feature at the beginning of February.

5

Thanks

6

Works, in that I can configure it without getting complaints about bad syntax, but I can't get the output I expect, also seems lots of lines are lost. Maybe it's still in alpha :slight_smile:

7

@seefood Multiline was release with 1.1. Can you give some more details on your issue?

8

Are grok patterns expanded?

I transformed my config from multiline codec that worked following this and this from:

codec => multiline {
      pattern => "^20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}"
      negate => true
      what => "previous"
    }

into:

multiline:
        pattern: "^20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}"
        negate: true
        match: after

and it's not splitting the log lines correctly. I see lines merged together when they shouldn't be. I also tried expanding the pattern into [0-9]{4}-[0-9]{2}-[0-9]{2}..., but it doesn't work neither.

9

Filebeat does not support grok patterns. For "valid" regexp see https://github.com/elastic/beats/issues/740

10

So I ended with

multiline:
        pattern: "^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}"
        negate: true
        match: after

and it seems to work OK. Thanks,

7 Likes
11

This should be written down in the documentation somewhere. I've been trying for hours to figure out what this achieves.

Thanks so much for posting your solution.

1 Like
12

@matthewcanty we're actually thinking about collecting and documenting configurations for different scenarios. Any multiline scenarios you are interested in (samples?) ?

13

This is the only one we've needed so far. The .Net stack trace:

2016-03-04 14:50:07,614 [16] ERROR MyService.Domain.WebApi.Controllers.GreetingsController - Error on saving greeting.
System.InvalidOperationException: Database does not exist
   at MyService.Persistence.NonMutatingDatabaseInitialiser`1.InitializeDatabase(TContext context)
   at System.Data.Entity.Internal.InternalContext.PerformInitializationAction(Action action)
   at System.Data.Entity.Internal.InternalContext.PerformDatabaseInitialization()
   at System.Data.Entity.Internal.RetryAction`1.PerformAction(TInput input)
   at System.Data.Entity.Internal.LazyInternalContext.InitializeDatabaseAction(Action`1 action)
   at System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType)
   at System.Data.Entity.Internal.Linq.InternalSet`1.Initialize()
   at System.Data.Entity.Internal.Linq.InternalSet`1.get_InternalContext()
   at System.Data.Entity.Infrastructure.DbQuery`1.System.Linq.IQueryable.get_Provider()
   at System.Linq.Queryable.Where[TSource](IQueryable`1 source, Expression`1 predicate)
   at MyService.Persistence.BaseRepository`2.Find(Expression`1 predicate)
   at MyService.Domain.ApplicationsServices.GreetingsService.<SaveGreetingsAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MyService.Domain.WebApi.Controllers.GreetingsController.<PostExternalAccounts>d__1.MoveNext()
2016-03-04 14:50:07,614 [16] ERROR MyService..........
2 Likes
14

@matthewcanty Thanks for providing your example. We will take it into account when updating our docs.
@dedemorton FYI

2 Likes
15

Thank you for posting this solution! This resolved my issue as well.

One of our applications writes logs multiline where all starting log entries start with the date itself. For example:

2017-02-09 04:00:21.517 C010 .... blah blah blah
2017-02-09 07:38:44.507 A5B4 .... we love all datas.... blah blah

The pattern you provided here works PERFECTLY. Thank you again @mmuszkow!