New line in file beat for java stack trace

kasi · April 4, 2016, 6:41pm

The following is my multi-line config (from file beat example from guide).

multiline:
    pattern: "^[[:space:]]+(at|...)|^Caused by:"
    negate: false
    match: after

This is the output in the log file

org.xml.sax.SAXParseException; lineNumber: 40; columnNumber: 40; SchemaLocation: schemaLocation value = 'http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc.xsd   http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.2.xsd   http://www.springframework.org/schema/data/repository http://www.springframework.org/schema/data/repository/spring-repository-1.6.xsd   http://cxf.apache.org/transports/camel http://cxf.apache.org/transports/camel.xsd   http://www.bnymellon.com/schema/xjp http://www.bnymellon.com/schema/xjp/beans.xsd   http://camel.apache.org/schema/cxf http://camel.apache.org/schema/cxf/camel-cxf.xsd   http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd   http://www.springframework.org/schema/data/jpa http://www.springframework.org/schema/data/jpa/spring-jpa.xsd   http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd   http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.bnymellon.com/schema/xjp ' must have even number of URI's.
        at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
        at org.apache.xerces.util.ErrorHandlerWrapper.warning(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.xs.XMLSchemaValidator.storeLocations(Unknown Source)
        at org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source)
        at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(Unknown Source)
        at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
        at org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
        at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:429)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:391)

This is not coming as a single event, it is being sent to logstash as multiple events..

If I were do this in log stash using the multiline codec is it possible?

Thanks,

Kasi

magnusbaeck · April 4, 2016, 8:28pm

Doesn't the log entry actually begin with a timestamp?

kasi · April 4, 2016, 11:16pm

Yes, it does, is there a test bed or shell where we can test the pattern and the log file entry to make
sure our multi-line pattern works

magnusbaeck · April 5, 2016, 5:30am

Then I suggest you set up your multiline pattern to join the current line with the previous line unless the line starts with a timestamp. There should be many examples of this floating around, both for Filebeat and Logstash (but beware that Logstash supports grok patterns in its regular expressions while Filebeat only supports pure regular expressions).

I think the closest you're going to get to a test bed is setting up a minimal Filebeat configuration that reads from stdin and emits to a console output.

steffens · April 5, 2016, 9:22am

can you add some more complete log output? Very hard to come up with regex if context is unclear/incomplete. When building regex for multiline it's very helpfull to see the full log-line (including timestamps) and even some lines before/after said event.

kasi · April 5, 2016, 9:56pm

19:09:46,536 [on(2)-127.0.0.1] INFO  ContextLoader                  - Root WebApplicationContext: initialization started
19:09:46,618 [on(2)-127.0.0.1] INFO  XmlWebApplicationContext       - Refreshing Root WebApplicationContext: startup date [Tue Apr 05 19:09:46 GMT 2016]; root of context hierarchy
19:09:46,650 [on(2)-127.0.0.1] INFO  XmlBeanDefinitionReader        - Loading XML bean definitions from class path resource [camel-config.xml]
19:09:46,901 [on(2)-127.0.0.1] WARN  XmlBeanDefinitionReader        - Ignored XML validation warning
org.xml.sax.SAXParseException; lineNumber: 40; columnNumber: 40; SchemaLocation: schemaLocation value = 'http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc.xsd   http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.2.xsd   http://www.springframework.org/schema/data/repository http://www.springframework.org/schema/data/repository/spring-repository-1.6.xsd   http://cxf.apache.org/transports/camel http://cxf.apache.org/transports/camel.xsd   http://www.bnymellon.com/schema/xjp http://www.bnymellon.com/schema/xjp/beans.xsd   http://camel.apache.org/schema/cxf http://camel.apache.org/schema/cxf/camel-cxf.xsd   http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd   http://www.springframework.org/schema/data/jpa http://www.springframework.org/schema/data/jpa/spring-jpa.xsd   http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd   http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.bnymellon.com/schema/xjp ' must have even number of URI's.
        at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
        at org.apache.xerces.util.ErrorHandlerWrapper.warning(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(Unknown Source)
        at org.apache.xerces.impl.xs.XMLSchemaValidator.storeLocations(Unknown Source)
        at org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source)
        at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(Unknown Source)
        at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
        at org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
        at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
        at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:429)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:391)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
        at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
        at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
        at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
        at org.springframework.web.context.support.XmlWebApplicationContext.loadBeanDefinitions(XmlWebApplicationContext.java:125)

        
        
19:09:47,145 [on(2)-127.0.0.1] INFO  XmlBeanDefinitionReader        - Loading XML bean definitions from class path resource [META-INF/cxf/cxf.xml]

andrewkroh · April 5, 2016, 10:30pm

I used the configuration below with Filebeat 5.0.0-alpha1 for my test bed. Each time I ran a test I deleted the .filebeat file from the current directory.

filebeat:
  prospectors:
    - paths: ["java.log"]
      multiline:
        pattern: '^[[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2},[[:digit:]]{3} '
        negate:  true
        match:   after
output:
  console:
    pretty: true
    bulk_max_size: 1
# For testing purposes limit the fields in the output.
filter:
  - include_fields:
      fields: [message]

kasi · April 6, 2016, 6:05pm

Thanks will try it out