Filebeat Multiline Weblogic//Java Stacktrace

JensK · August 18, 2016, 1:42pm

Hello guys

im testing ELK with Weblogic Application Server and want to parse the server.out file with filebeat.

This ist my Example file....

<18.08.2016 14:00 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
<18.08.2016 14:01 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
<18.08.2016 14:02 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
<18.08.2016 14:03 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
        at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
>

Im using this filebeat.yml

paths:
- /domains/weblogic-test/servers/*/*/*.out
encoding: plain
input_type: log
document_type: wl_test
scan_frequency: 0s
multiline:
        pattern: '^\<[[:digit:]]{2}.[[:digit:]]{2}.[[:digit:]]{4} [[:digit:]]{2}:[[:digit:]]{2} Uhr MESZ'
        negate: true
        match: after
        max_lines: 10000
        timeout: 10s

So now sometime i got this result - but not everytime ... i think in 9 out of 10 cases the Filebeat parses the File correctly!!! Does someone got any idea?

MSG 1: <18.08.2016 14:00 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
MSG 2: <18.08.2016 14:02 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration>
MSG 3:<18.08.2016 14:03 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)

MSG 4  javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
        Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
        at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
>

greets Jens

ruflin · August 19, 2016, 6:40am

Which filebeat version are you using? If 1.x here is a tool that you can use to test with your messages: https://www.elastic.co/guide/en/beats/filebeat/1.2/multiline-examples.html

Same for the 5.0 release: https://www.elastic.co/guide/en/beats/filebeat/5.0/multiline-examples.html

Some of the pattern changed. I haven't checked your pattern in detail but my assumption is that it does only match 9 our of 10 events. Would be nice to see the event it didn't match.

JensK · August 19, 2016, 7:04am

Hello Rufin, thanks for your help.

I checked the pattern with the go Playground an a bunch of messages. All seems to be fine.
Actually i think that its no problem of the pattern, cause the same message works sometime and somtime the message (java stacktrace with weblogic header) is splitted in many messages.

Maybe my Filebeat multiline config ist wrong, but i didnt get what i configured wrong ...
I had the same problem as i tried it with logstash multiline filter...

btw. i parse another filetype with an second prospector wich contains nearly the same messages (weblogic header, java stacktrace) and there it works fine ... (there is an difference in the Pattern cause the message dont start with an "<")

greets

steffens · August 19, 2016, 2:42pm

have you some more samples with lines being correctly merged and where it fails? Would like to give it a try with go playground. Having some more samples will help checking pattern doesn't break anything else.

JensK · August 19, 2016, 3:51pm

Dont Working:

1# split at line 11

<19.08.2016 17:23 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
	>

2# split at line 13

<19.08.2016 17:31 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
>

JensK · August 19, 2016, 3:51pm

3# Split at Line 12

	<19.08.2016 16:28 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
>

JensK · August 19, 2016, 3:51pm

worked:
1#

<19.08.2016 17:26 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
	>

2#

<19.08.2016 17:25 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration> 
3#
<19.08.2016 17:30 Uhr MESZ> <Alert> <Security> <BEA-090717> <Invalid server AdminServer SSL configuration> 
4#
<19.08.2016 17:33 Uhr MESZ> <Warning> <RMI> <BEA-080003> <A RuntimeException was generated by the RMI server: javax.management.remote.rmi.RMIConnectionImpl.getAttribute(Ljavax.management.ObjectName;Ljava.lang.String;Ljavax.security.auth.Subject;)
 javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException.
javax.management.RuntimeMBeanException: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	Truncated. see log file for complete stacktrace
Caused By: javax.management.RuntimeMBeanException: MBean getAttribute failed: java.lang.NullPointerException
	at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:553)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
	at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:336)
	at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$12.run(WLSMBeanServerInterceptorBase.java:326)
	Truncated. see log file for complete stacktrace
Caused By: java.lang.NullPointerException
	at weblogic.socket.JSSEFilterImpl.<init>(JSSEFilterImpl.java:58)
	at weblogic.socket.JSSESocketFactory.getJSSEFilterImpl(JSSESocketFactory.java:97)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:49)
	at weblogic.socket.JSSESocketFactory.createSocket(JSSESocketFactory.java:92)
	at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:81)
	Truncated. see log file for complete stacktrace
>

JensK · August 19, 2016, 3:54pm

hmm i noticed yet that there are sometime some messages between them ... :-/
How is this possible? Filebeat scans one logfile for me, an he splits one message in two parts. But before he send the second part he decides to send an completly other message?
Dont get it...

ruflin · August 22, 2016, 8:19am

You mean it sends a log message from an other log file in between?

About your log messages: Is filebeat only sending the first part of the multiline event or do you get 2 messages in ES in the end? Are all your messages written by your system at once? Means it could it be that a timeout is happening in the middle of the event?

JensK · August 22, 2016, 8:37am

From the same Logfile - i diasbled all other prospector atm, so i can be sure that the event ist from AdminServer.out

I got 2 (or more) Messages in ES.
The theorie with an Timeout is possible... but die Same Events are written too Adminserver.log (dont know why Weblogic are providse 2 times the same log ) an the prospector for Adminserver.log works fine....
The difference is that every Message on Adminserver.log starts with "####" so i chosed the pattern '^####'

steffens · August 22, 2016, 10:13am

I see no problems with the regex. you might consider shortening the pattern a little though. It seems ^\< should do the trick already. All samples do work fine with regex either way.

For events becoming split up I see 2 potential causes:

multiple threads writing to log-file at same time. If logging is not fully atomic on message level events might become mixed up.
output buffering. Depending on the loggers buffering strategy logs might be only written after buffer has reached a limit and or some internal timeout triggers the flushing (e.g. fwrite - C API - is normally buffering output in user process). One option to overcome potential problems with buffering is adapt the timeout in multiline-config. If all events do always end with a new-line character I would even disable the timeout on multiline and see how it goes.

Please note, if multiline timeout is disabled, the latency (time you have logs available) is also subject to log producing process.

Another option is to only send/index full or event-beginnings (remove 2, 3rd... parts of incomplete log). This way the timeout option on multiline will make latency more predictable (still subject to buffering strategy, though) at the cost of having incomplete stack-traces. This can be achieved by setting include_lines: ['^\<'] as well.

system · September 8, 2016, 1:42pm

This topic was automatically closed after 21 days. New replies are no longer allowed.