Filebeat-Multiline

SJN8 · August 23, 2018, 11:21am

Hi All,

I have below logs pattern but not able to club them in multiline ,

Aug 23, 2018 2:38:23 AM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Aug 23, 2018 2:38:23 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-7023"]
Aug 23, 2018 2:38:23 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service test
Aug 23, 2018 2:38:23 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/ROOT
Aug 23, 2018 2:38:24 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/tomee
Aug 23, 2018 2:38:24 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/host-manager
Aug 23, 2018 2:38:24 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/healthcheck
Aug 23, 2018 2:38:24 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/manager
Aug 23, 2018 2:38:25 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/pcs
Aug 23, 2018 2:38:25 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/context
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-2] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-3] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-4] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-5] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-6] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-7] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-8] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-9] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-10] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-11] but has failed to stop it. This is very likely to create a memory leak.
Aug 23, 2018 2:38:25 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-12] 
INFO: Stopped ActiveMQ broker
Aug 23, 2018 2:38:25 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-7023"]
Aug 23, 2018 2:59:03 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
Aug 23, 2018 2:59:03 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-7023"]
Aug 23, 2018 2:59:03 AM org.apache.openejb.util.OptionsLog info
INFO: Using 'openejb.jdbc.datasource-creator=org.apache.tomee.jdbc.TomEEDataSourceCreator'
Aug 23, 2018 2:59:03 AM org.apache.openejb.OpenEJB$Instance <init>
INFO: ********************************************************************************
Aug 23, 2018 2:59:03 AM org.apache.openejb.OpenEJB$Instance <init>
INFO: OpenEJB http://tomee.apache.org/
Aug 23, 2018 2:59:03 AM org.apache.openejb.OpenEJB$Instance <init>
INFO: Startup: Thu Aug 23 02:59:03 CEST 2018
Aug 23, 2018 2:59:03 AM org.apache.openejb.OpenEJB$Instance <init>
INFO: Copyright 1999-2013 (C) Apache OpenEJB Project, All Rights Reserved.
Aug 23, 2018 2:59:03 AM org.apache.openejb.OpenEJB$Instance <init>
INFO: Version: 4.7.2
Aug 23, 2018 2:59:03 AM org.apache.openejb.OpenEJB$Instance <init>
INFO: Build date: 20150517
Aug 23, 2018 2:59:03 AM org.apache.openejb.OpenEJB$Instance <init>

my configuration,
multiline.pattern: '^[A-Za-z]{3}|^[A-Za-z]{4}|^[A-Za-z]{6}'
multiline.negate: true
multiline.match: before
multiline.max_lines: 2000

i have tried with negate False/True , match after/before ,

can someone help me what is wrong with above settings ?

i have tired with [[:alpha:]] in place of [A-Za-z] but that also did not worked for me .

Thanks,
Shashank

steffens · August 27, 2018, 1:42pm

Only add a pattern for dates to 'pattern'. You want to collect all lines not starting with a date.

e.g. '^[A-Z][a-z]{2}, \d{4} \d:\d\d:\d\d [AP]M

SJN8 · August 28, 2018, 5:56am

@steffens
no i want to collect line i a group , like below

Aug 23, 2018 2:38:23 AM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Aug 23, 2018 2:38:23 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-7023"]
Aug 23, 2018 2:38:23 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service test
Aug 23, 2018 2:38:23 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/ROOT
Aug 23, 2018 2:38:24 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/tomee
Aug 23, 2018 2:38:24 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
INFO: Undeploying app: /usr/local/app/apache-tomee-plus-1.7.2/webapps/host-manager
Aug 23, 2018 2:38:24 AM org.apache.openejb.assembler.classic.Assembler destroyApplication
SEVERE: The web application [/context] appears to have started a thread named [pool-13-thread-12] 


as i have set max line as 2000 , will change it to 200 or 300 but all line should be as a single even so i am trying to form a pattern to combine  lines starting with  
Aug 23, 2018 2:38:23 AM or INFO: or SEVERE:  

but as you suggested above pattern would work only for lines starting with date not with other two . 

please suggest a way for above requirement . 

Thanks, 
Shashank

steffens · August 28, 2018, 2:07pm

Does the file contain more logs? Like logs before or after you want to have as separate events?

Is your ask to send the complete file as one event? Do you have some kind of condition to split events?

The multiline feature will collect all matching lines. If max_lines is less then the actual line count, the the extra lines will be removed from the event. The max_lines setting does not configure a 'split'.

SJN8 · August 31, 2018, 6:43am

@steffens ,
yes file contains more , and i can send events in a group of 100-200 lines .
split events if it exceeds the max line a event like 100-200 .

system · September 28, 2018, 6:43am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Need help for the correct multline pattern Beats filebeat	2	353	May 7, 2018
Multiple Multiline Options Beats filebeat	7	7091	August 9, 2017
Filebeat multiline Pattern on Apache Logs does not work as expected (Filebeat -> Logstash -> ES) Beats filebeat	3	373	May 5, 2022
Problem with Tomcat multiline entries Beats filebeat	2	2314	June 29, 2017
Multiline parser filebeat 6.4.2 Beats filebeat	5	498	November 22, 2018

Filebeat-Multiline

Related topics