Filebeat multiline regexp patterns help

Elastic Stack Beats
1

Hi,

I'm trying to use multline in filebeat to ship this application log entry as 1 document to logstash.

2019-02-12 00:01:57,302 WARN  [ScheduledTask-1] 
[name=account@domain.com;mid=1513;ds=CAL-11170;] datasource - Scheduled DataSource import failed.
com.zimbra.common.service.ServiceException: resource unreachable: IOException: java.net.SocketTimeoutException: connect timed out
ExceptionId:ScheduledTask-1:1549926117299:669654c801f05003
Code:service.RESOURCE_UNREACHABLE
    at com.zimbra.common.service.ServiceException.RESOURCE_UNREACHABLE(ServiceException.java:310)
    at com.zimbra.cs.service.FeedManager.retrieveRemoteDatasource(FeedManager.java:360)
    at com.zimbra.cs.mailbox.Mailbox.importFeed(Mailbox.java:8409)
    at com.zimbra.cs.mailbox.Mailbox.synchronizeFolder(Mailbox.java:8398)
    at com.zimbra.cs.datasource.RssImport.importData(RssImport.java:54)
    at com.zimbra.cs.datasource.DataSourceManager.importData(DataSourceManager.java:403)
    at com.zimbra.cs.datasource.DataSourceManager.importData(DataSourceManager.java:359)
    at com.zimbra.cs.datasource.DataSourceTask.call(DataSourceTask.java:83)
    at com.zimbra.cs.datasource.DataSourceTask.call(DataSourceTask.java:31)
    at com.zimbra.common.util.TaskScheduler$TaskRunner.call(TaskScheduler.java:79)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at com.zimbra.common.net.ProtocolSocketFactoryWrapper.createSocket(ProtocolSocketFactoryWrapper.java:48)
    at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
    at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
    at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
    at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
    at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:61)
    at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:47)
    at com.zimbra.cs.service.FeedManager.retrieveRemoteData(FeedManager.java:254)
    at com.zimbra.cs.service.FeedManager.retrieveRemoteDatasource(FeedManager.java:348)
    ... 14 more

For doing it, I was trying to follow these articles:

This is my multiline conf (one of the dozens I've tried :frowning: )in the filebeats.yml.
multiline.pattern: '\Q"Scheduled DataSource import failed."\E'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: '[[:space:]]*(.){3}[[:space:]]more'

In this case, it justs matches a lot of things, a lot more than I expected (other log entries, or in this particular case it will match things that go above and below the log sample I provide here). I'm not understanding how to use these regexs. If I try patterns, I can get the expected results in here https://play.golang.org/ but when I try them in filebeats, I just can't make them work as expected.

What worries me is that this is not even the final form of what I need, as after I get this multiline right, I will need to match several other multiline patrons/texts and I can't seem to find many examples of multiline with filebeat.

Any help is more than welcome.

2

Hi,

We use the following config in FIleBeat for our java logs:
multiline.pattern: ^\d\d\d\d-\d\d-\d\d
multiline.negate: true
multiline.match: after

This creates a new entry each time a timestamp is found, the rest of the lines are appended to the last log entry.

3

Hi Wolfram!

That worked perfectly. Even better, your example allowed me to better understand the examples and explanaitions given in the official documentation, so I think that I will be able to keep the log parsing rolling :slight_smile:

Thank you very much for your help!

4

@Wolfram_Haussig Do you mind if I add your example to the docs?

5

Of cours not. I am glad to help.