Multiline codec in Filebeat is not working

Elastic Stack Beats
1

I am trying to parse catalina.out file using Logstash. All my logs starts with an timestamp field like : 2017-06-20 07:12:55.226

I tried couple of multiline patterns in filebeat yml but nothing seems to be working:

pattern: "[1]{4}-[[:digit:]]{2}-[[:digit:]]{2} "

pattern: '^\d{4}-\d{2}-\d{2} '

pattern: "[2]{4}-[:digit:]{2}-[:digit:]{2} "

pattern: "[3]{4}-[0-9]{2}-[0-9]{2} "

here is my logstash.conf:

filter {
 if [type] == "mina-log"
     {
     grok {
            match => ["message", "(?<logtimestamp>%{DATE}%{SPACE}%{TIME})%{SPACE}\[%{LOGLEVEL:loglevel}%{SPACE}\]%{SPACE}\[%{NOTSPACE:module}\]%{SPACE}%{NOTSPACE:applicationname}%{SPACE}\-%{SPACE}(?<logmessage>(.|\n)+)"]
          }
     date {
         locale => "en"
         match => ["logtimestamp", "YY-MM-dd HH:mm:ss.SSS"]
 	timezone => "America/New_York"
         #timezone => "UTC"
         target => "@logtimestamp"
         }
     }    
     }

FileBeat yml:

filebeat:
  prospectors:
    -
      paths:
        - D:\ELK\filebeat\source\catalina.out
      document_type: mina-log
  multiline:
        pattern: '^\d{4}-\d{2}-\d{2} '
        negate: true
        match: after

Data Input:

2017-06-16 03:51:51.191 [ERROR] [localhost-startStop-1] SpringApplication - Application startup failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'advSearchController': Unsatisfied dependency expressed through field 'codeUtil'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'codeUtil': Unsatisfied dependency expressed through field 'lCodeDecode'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'localCodeDecodeImpl': Unsatisfied dependency expressed through field 'jdbcTemplate'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jdbcTemplate' defined in class path resource [cefcontext.xml]: Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in class path resource [cefcontext.xml]: Invocation of init method failed; nested exception is javax.naming.NamingException: Unable to load class: oracle.jdbc.OracleDriver from ClassLoader:java.net.URLClassLoader@548c4f57;ClassLoader:ParallelWebappClassLoader
context: minawebapp
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@548c4f57

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.7.RELEASE.jar:4.3.7.RELEASE]

2017-06-20 07:12:55.226 [INFO ] [localhost-startStop-1] MinawebappApplication - No active profile set, falling back to default profiles: default

For the about 2 input lines - the logstash is parsing only first line of each input. But I want to get all the data between "Application startup failed" and "RELEASE]" with in first log message

  1. [:digit:] ↩︎

  2. :digit: ↩︎

  3. 0-9 ↩︎

2

However, adding the following codec in Logstash seems to be working though:

     codec => multiline {
  	     pattern => '^\d{4}-\d{2}-\d{2} '
         negate => true
         what => "previous"
		        }

Not sure why similar code is not working in Filebeat

3

Have you tried testing your patterns using the Go Playground as described on this page?

4

Also, double check the indentation of your config file. The multiline config options are part of the prospector config so they should have the same indentation level as the paths option.

filebeat:
  prospectors:
    -
      paths:
        - D:\ELK\filebeat\source\catalina.out
      document_type: mina-log
      multiline:
        pattern: '^\d{4}-\d{2}-\d{2} '
        negate: true
        match: after
5

Hi Andrew - thanks for your response. YES the indentation was the issue, now it is parsing after correcting the indent as per your suggestion!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.