Hi,
I'm trying to use multline in filebeat to ship this application log entry as 1 document to logstash.
2019-02-12 00:01:57,302 WARN [ScheduledTask-1]
[name=account@domain.com;mid=1513;ds=CAL-11170;] datasource - Scheduled DataSource import failed.
com.zimbra.common.service.ServiceException: resource unreachable: IOException: java.net.SocketTimeoutException: connect timed out
ExceptionId:ScheduledTask-1:1549926117299:669654c801f05003
Code:service.RESOURCE_UNREACHABLE
at com.zimbra.common.service.ServiceException.RESOURCE_UNREACHABLE(ServiceException.java:310)
at com.zimbra.cs.service.FeedManager.retrieveRemoteDatasource(FeedManager.java:360)
at com.zimbra.cs.mailbox.Mailbox.importFeed(Mailbox.java:8409)
at com.zimbra.cs.mailbox.Mailbox.synchronizeFolder(Mailbox.java:8398)
at com.zimbra.cs.datasource.RssImport.importData(RssImport.java:54)
at com.zimbra.cs.datasource.DataSourceManager.importData(DataSourceManager.java:403)
at com.zimbra.cs.datasource.DataSourceManager.importData(DataSourceManager.java:359)
at com.zimbra.cs.datasource.DataSourceTask.call(DataSourceTask.java:83)
at com.zimbra.cs.datasource.DataSourceTask.call(DataSourceTask.java:31)
at com.zimbra.common.util.TaskScheduler$TaskRunner.call(TaskScheduler.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.zimbra.common.net.ProtocolSocketFactoryWrapper.createSocket(ProtocolSocketFactoryWrapper.java:48)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:61)
at com.zimbra.common.httpclient.HttpClientUtil.executeMethod(HttpClientUtil.java:47)
at com.zimbra.cs.service.FeedManager.retrieveRemoteData(FeedManager.java:254)
at com.zimbra.cs.service.FeedManager.retrieveRemoteDatasource(FeedManager.java:348)
... 14 more
For doing it, I was trying to follow these articles:
- https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
- https://www.elastic.co/guide/en/beats/filebeat/current/_examples_of_multiline_configuration.html
- https://www.elastic.co/guide/en/beats/filebeat/current/regexp-support.html#grouping
This is my multiline conf (one of the dozens I've tried )in the filebeats.yml.
multiline.pattern: '\Q"Scheduled DataSource import failed."\E'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: '[[:space:]]*(.){3}[[:space:]]more'
In this case, it justs matches a lot of things, a lot more than I expected (other log entries, or in this particular case it will match things that go above and below the log sample I provide here). I'm not understanding how to use these regexs. If I try patterns, I can get the expected results in here https://play.golang.org/ but when I try them in filebeats, I just can't make them work as expected.
What worries me is that this is not even the final form of what I need, as after I get this multiline right, I will need to match several other multiline patrons/texts and I can't seem to find many examples of multiline with filebeat.
Any help is more than welcome.