Filebeat multiline

Elastic Stack Beats
1

Hi all,

i have the following log file

Start time Wednesday, July 12, 2017 07:34:19 PM SGT by murex

File descriptors raised to 16384 for current cmd.

Java option used :  -d64  -showversion  -Djavax.xml.stream.XMLEventFactory=com.ctc.wstx.stax.WstxEventFactory -Djavax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory -Djavax.xml.stream.XMLOutputFactory=com.ctc.wstx.stax.WstxOutputFactory -XX:OnOutOfMemoryError=./kill3.sh -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/murex/UAT337/mx/heapdumps -Xmx4G -verbose:gc -XX:+PrintGCTimeStamps -Xloggc:logs/gc/xmls.20170712193403.gc.log 
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)


Java cmd:
java -Xms32M -Xmx64M  -server  -d64  -showversion  -Djavax.xml.stream.XMLEventFactory=com.ctc.wstx.stax.WstxEventFactory -Djavax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory -Djavax.xml.stream.XMLOutputFactory=com.ctc.wstx.stax.WstxOutputFactory -XX:OnOutOfMemoryError=./kill3.sh -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/murex/UAT337/mx/heapdumps -Xmx4G -verbose:gc -XX:+PrintGCTimeStamps -Xloggc:logs/gc/xmls.20170712193403.gc.log  -cp mxjboot.jar -Djava.rmi.server.codebase=http://s01smxu10z4a.sgp.dbs.com:13370/murex.download.service.download murex.rmi.loader.RmiLoader /MXJ_CLASS_NAME:murex.apps.middleware.server.home.XmlHomeStartAll /MXJ_SITE_NAME:site1 /MXJ_LOGGER_FILE:public.mxres.loggers.default_logger.mxres /MXJ_HUB_NAME:hub1.site1   

java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

mxjboot.jar  Build-ID: 3441713-160315-1653-1744129
Opening jar/file.version...
Opening bin/file.version...
Checking files...
jar/file.version saved.
bin/file.version saved.
2017-07-12 19:34:27,355 [main] console : INFO - Loading public/mxres/common/rights.mxres from file server http://s01smxu10z4a.sgp.dbs.com:13370/murex.download.service.download
2017-07-12 19:34:27,478 [main] console : INFO - XmlHome [[ID:-1186478599][ConnectionsAddress:[UserName:murex][HostName:s01smxu10z4a][IPAddress:10.92.131.168][SiteName:site1][ClientRPCTransportConfig:[Protocol:RMI][Site:site1][ServerName:XMLS][Host:s01smxu10z4a.sgp.dbs.com][Port:13372][Connectable:Y]]][RegistryServerInfo : [ID:-1186478599][ConnectionsAddress:[UserName:murex][HostName:s01smxu10z4a][IPAddress:10.92.131.168][SiteName:site1][ClientRPCTransportConfig:[Protocol:RMI][Site:site1][ServerName:XMLS][Host:s01smxu10z4a.sgp.dbs.com][Port:13372][Connectable:Y]][[CreationTime:Wed Jul 12 19:34:27 SGT 2017][Type:MX-XML Server][Description:XmlHome Server][Source:null]]]] started.
2017-07-12 19:34:27,478 [main] console : INFO - Server ready...
2017-07-12 19:34:27,488 [main] console : INFO - Loading public/mxres/loggers/default_logger.mxres from file server http://s01smxu10z4a.sgp.dbs.com:13370/murex.download.service.download
2017-07-12 19:34:27,491 [main] console : INFO - Loading public/mxres/loggers/common_logger.mxres from file server http://s01smxu10z4a.sgp.dbs.com:13370/murex.download.service.download
2017-07-12 19:34:27,586 [main] console : INFO - HubHome [[ID:-1077833616][ConnectionsAddress:[UserName:murex][HostName:s01smxu10z4a][IPAddress:10.92.131.168][SiteName:site1][ClientRPCTransportConfig:[Protocol:RMI][Site:site1][ServerName:HUB][Host:s01smxu10z4a.sgp.dbs.com][Port:13373]][HubName:hub1]][HubInfo : [HubName:null][ID:-1077833616][ConnectionsAddress:[UserName:murex][HostName:s01smxu10z4a][IPAddress:10.92.131.168][SiteName:site1][ClientRPCTransportConfig:[Protocol:RMI][Site:site1][ServerName:HUB][Host:s01smxu10z4a.sgp.dbs.com][Port:13373]][[CreationTime:Wed Jul 12 19:34:27 SGT 2017][Type:MX-XML Server][Description:HubHome Server [hub1]][Source:null]]]] started.
2017-07-12 19:34:27,587 [main] console : INFO - Server ready...
2017-07-12 19:35:21,752 [Thread-11] console : INFO - Loading license/MXINTERNAL_LOGINIMPL_customerrights.mxres from file server http://s01smxu10z4a.sgp.dbs.com:13370/murex.download.service.download
2017-07-12 19:35:22,621 [Thread-9] console : INFO - Loading license/MXCONTRIBUTION_customerrights.mxres from file server http://s01smxu10z4a.sgp.dbs.com:13370/murex.download.service.download
2017-07-12 19:35:23,616 [Thread-6] console : INFO - Loading license/MXFEDERATION_customerrights.mxres from file server http://s01smxu10z4a.sgp.dbs.com:13370/murex.download.service.download
2017-07-12 19:36:17,366 [Thread-9] console : INFO - Loading license/MXDICTIONARY_customerrights.mxres from file server http://s01smxu10z4a.sgp.dbs.com:13370/murex.download.service.download

I am using the following multiline conf for filebeat but it doesn't seem to break up the data properly.

  -
    document_type: coreservices-xmls
    tags: ["launcher"]
    paths:
      - /appdir/logs/*.xmlserver.*.log
    multiline:
      pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'|'Start'|'Process'
      negate: true
      match: after

it breaks at 'Start' and 'Process' but it does not separate out the lines that begins with dates..

I can't seem to find out where i went wrong.

Thanks

2 
^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'

This seems to assume the timestamp is starting with [, which is not the case in your file. Or am I misreading that?

3

wow im so careless

thanks so much

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.