Filebeat Multiline Weblogic//Java Stacktrace


(Jens Krämer) #1

Hello guys :slight_smile:

im testing ELK with Weblogic Application Server and want to parse the server.out file with filebeat.

This ist my Example file....

<18.08.2016 14:00 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
<18.08.2016 14:01 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
<18.08.2016 14:02 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
<18.08.2016 14:03 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
        at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
>

Im using this filebeat.yml

paths:
- /domains/weblogic-test/servers/*/*/*.out
encoding: plain
input_type: log
document_type: wl_test
scan_frequency: 0s
multiline:
        pattern: '^\<[[:digit:]]{2}.[[:digit:]]{2}.[[:digit:]]{4} [[:digit:]]{2}:[[:digit:]]{2} Uhr MESZ'
        negate: true
        match: after
        max_lines: 10000
        timeout: 10s

So now sometime i got this result - but not everytime ... i think in 9 out of 10 cases the Filebeat parses the File correctly!!! Does someone got any idea?

MSG 1: <18.08.2016 14:00 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
MSG 2: <18.08.2016 14:02 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
MSG 3:<18.08.2016 14:03 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)

MSG 4  javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
        Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
        at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
>

greets Jens


(ruflin) #2

Which filebeat version are you using? If 1.x here is a tool that you can use to test with your messages: https://www.elastic.co/guide/en/beats/filebeat/1.2/multiline-examples.html

Same for the 5.0 release: https://www.elastic.co/guide/en/beats/filebeat/5.0/multiline-examples.html

Some of the pattern changed. I haven't checked your pattern in detail but my assumption is that it does only match 9 our of 10 events. Would be nice to see the event it didn't match.


(Jens Krämer) #3

Hello Rufin, thanks for your help.

I checked the pattern with the go Playground an a bunch of messages. All seems to be fine.
Actually i think that its no problem of the pattern, cause the same message works sometime and somtime the message (java stacktrace with weblogic header) is splitted in many messages.

Maybe my Filebeat multiline config ist wrong, but i didnt get what i configured wrong ...
I had the same problem as i tried it with logstash multiline filter...

btw. i parse another filetype with an second prospector wich contains nearly the same messages (weblogic header, java stacktrace) and there it works fine ... (there is an difference in the Pattern cause the message dont start with an "<")

greets :slight_smile:


(Steffen Siering) #4

have you some more samples with lines being correctly merged and where it fails? Would like to give it a try with go playground. Having some more samples will help checking pattern doesn't break anything else.


(Jens Krämer) #5

Dont Working:

1# split at line 11

<19.08.2016 17:23 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
	> 

2# split at line 13

<19.08.2016 17:31 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
>

(Jens Krämer) #6

3# Split at Line 12

	<19.08.2016 16:28 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
>

(Jens Krämer) #7

worked:
1#

<19.08.2016 17:26 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
	> 

2#

<19.08.2016 17:25 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration> 
3#
<19.08.2016 17:30 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration> 
4#
<19.08.2016 17:33 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
>

(Jens Krämer) #8

hmm i noticed yet that there are sometime some messages between them ... :-/
How is this possible? Filebeat scans one logfile for me, an he splits one message in two parts. But before he send the second part he decides to send an completly other message?
Dont get it...


(ruflin) #9

You mean it sends a log message from an other log file in between?

About your log messages: Is filebeat only sending the first part of the multiline event or do you get 2 messages in ES in the end? Are all your messages written by your system at once? Means it could it be that a timeout is happening in the middle of the event?


(Jens Krämer) #10

From the same Logfile - i diasbled all other prospector atm, so i can be sure that the event ist from AdminServer.out

I got 2 (or more) Messages in ES.
The theorie with an Timeout is possible... but die Same Events are written too Adminserver.log (dont know why Weblogic are providse 2 times the same log :wink: ) an the prospector for Adminserver.log works fine....
The difference is that every Message on Adminserver.log starts with "####" so i chosed the pattern '^####' :slight_smile:


(Steffen Siering) #11

I see no problems with the regex. you might consider shortening the pattern a little though. It seems ^\< should do the trick already. All samples do work fine with regex either way.

For events becoming split up I see 2 potential causes:

  • multiple threads writing to log-file at same time. If logging is not fully atomic on message level events might become mixed up.
  • output buffering. Depending on the loggers buffering strategy logs might be only written after buffer has reached a limit and or some internal timeout triggers the flushing (e.g. fwrite - C API - is normally buffering output in user process). One option to overcome potential problems with buffering is adapt the timeout in multiline-config. If all events do always end with a new-line character I would even disable the timeout on multiline and see how it goes.

Please note, if multiline timeout is disabled, the latency (time you have logs available) is also subject to log producing process.

Another option is to only send/index full or event-beginnings (remove 2, 3rd... parts of incomplete log). This way the timeout option on multiline will make latency more predictable (still subject to buffering strategy, though) at the cost of having incomplete stack-traces. This can be achieved by setting include_lines: ['^\<'] as well.


(system) #12

This topic was automatically closed after 21 days. New replies are no longer allowed.